[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:40:47 dist-test-slave-csnl krb5kdc[2354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:40:47 dist-test-slave-csnl krb5kdc[2354](info): set up 2 sockets
Sep 25 22:40:47 dist-test-slave-csnl krb5kdc[2354](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 22:40:48.667917 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:40:48 dist-test-slave-csnl krb5kdc[2354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:40:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:40:48.802196 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:41703
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:33017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:41703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:48.923630 2367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:48.923982 2367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:48.924053 2367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:48.928401 2367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:48.928526 2367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:48.928570 2367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:48.928607 2367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:48.928642 2367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:48.934516 2367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:33017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:41703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:41703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:48.935794 2367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:48.936991 2367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:48.946718 2373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:48.946772 2372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:48.946887 2367 server_base.cc:1034] running on GCE node
W20260925 22:40:48.946774 2375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:48.947753 2367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:48.949090 2367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:48.950294 2367 hybrid_clock.cc:648] HybridClock initialized: now 1790376048950252 us; error 58 us; skew 500 ppm
Sep 25 22:40:48 dist-test-slave-csnl krb5kdc[2354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376048, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:48.953409 2367 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:40:48.954437 2367 webserver.cc:460] Webserver started at http://127.2.74.254:41175/ using document root <none> and password file <none>
I20260925 22:40:48.955119 2367 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:48.955197 2367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:48.955384 2367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:48.957641 2367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "58802a6c852e49efb7b1083255015618"
format_stamp: "Formatted at 2026-09-25 22:40:48 on dist-test-slave-csnl"
server_key: "7584ba17f0f66e631b70792a758d282e"
server_key_iv: "2490d1109d4f352beda2019f2ef276ef"
server_key_version: "encryptionkey@0"
I20260925 22:40:48.958226 2367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "58802a6c852e49efb7b1083255015618"
format_stamp: "Formatted at 2026-09-25 22:40:48 on dist-test-slave-csnl"
server_key: "7584ba17f0f66e631b70792a758d282e"
server_key_iv: "2490d1109d4f352beda2019f2ef276ef"
server_key_version: "encryptionkey@0"
I20260925 22:40:48.963075 2367 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 22:40:48.965840 2382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:48.967347 2367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:40:48.967490 2367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "58802a6c852e49efb7b1083255015618"
format_stamp: "Formatted at 2026-09-25 22:40:48 on dist-test-slave-csnl"
server_key: "7584ba17f0f66e631b70792a758d282e"
server_key_iv: "2490d1109d4f352beda2019f2ef276ef"
server_key_version: "encryptionkey@0"
I20260925 22:40:48.967613 2367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:48.985921 2367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:48.989367 2367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:48.989569 2367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:48.998039 2367 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:41703
I20260925 22:40:48.998064 2434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:41703 every 8 connection(s)
I20260925 22:40:48.999325 2367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:49.002581 2435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:49.009524 2435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58802a6c852e49efb7b1083255015618: Bootstrap starting.
I20260925 22:40:49.009936 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 2367
I20260925 22:40:49.010118 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:40:49.011746 2347 external_mini_cluster.cc:1428] Setting key 5fae903ddadc4449315a53005fa70204
I20260925 22:40:49.013778 2435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58802a6c852e49efb7b1083255015618: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:49.015372 2435 log.cc:824] T 00000000000000000000000000000000 P 58802a6c852e49efb7b1083255015618: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:49.017884 2435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58802a6c852e49efb7b1083255015618: No bootstrap required, opened a new log
I20260925 22:40:49.022388 2435 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58802a6c852e49efb7b1083255015618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58802a6c852e49efb7b1083255015618" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 41703 } }
I20260925 22:40:49.022729 2435 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58802a6c852e49efb7b1083255015618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:49.022789 2435 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58802a6c852e49efb7b1083255015618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58802a6c852e49efb7b1083255015618, State: Initialized, Role: FOLLOWER
I20260925 22:40:49.023370 2435 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58802a6c852e49efb7b1083255015618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58802a6c852e49efb7b1083255015618" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 41703 } }
I20260925 22:40:49.023538 2435 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58802a6c852e49efb7b1083255015618 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:40:49.023625 2435 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58802a6c852e49efb7b1083255015618 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:40:49.023746 2435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58802a6c852e49efb7b1083255015618 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:49.024785 2435 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58802a6c852e49efb7b1083255015618 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58802a6c852e49efb7b1083255015618" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 41703 } }
I20260925 22:40:49.025173 2435 leader_election.cc:302] T 00000000000000000000000000000000 P 58802a6c852e49efb7b1083255015618 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58802a6c852e49efb7b1083255015618; no voters:
I20260925 22:40:49.025557 2435 leader_election.cc:288] T 00000000000000000000000000000000 P 58802a6c852e49efb7b1083255015618 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:40:49.025697 2440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58802a6c852e49efb7b1083255015618 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:49.025955 2440 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58802a6c852e49efb7b1083255015618 [term 1 LEADER]: Becoming Leader. State: Replica: 58802a6c852e49efb7b1083255015618, State: Running, Role: LEADER
I20260925 22:40:49.026309 2440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58802a6c852e49efb7b1083255015618 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58802a6c852e49efb7b1083255015618" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 41703 } }
I20260925 22:40:49.027155 2435 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58802a6c852e49efb7b1083255015618 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:49.030313 2442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58802a6c852e49efb7b1083255015618 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58802a6c852e49efb7b1083255015618. Latest consensus state: current_term: 1 leader_uuid: "58802a6c852e49efb7b1083255015618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58802a6c852e49efb7b1083255015618" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 41703 } } }
I20260925 22:40:49.030470 2442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58802a6c852e49efb7b1083255015618 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:49.031121 2441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58802a6c852e49efb7b1083255015618 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58802a6c852e49efb7b1083255015618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58802a6c852e49efb7b1083255015618" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 41703 } } }
I20260925 22:40:49.031250 2441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58802a6c852e49efb7b1083255015618 [sys.catalog]: This master's current role is: LEADER
W20260925 22:40:49.033236 2454 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 58802a6c852e49efb7b1083255015618: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:40:49.033337 2454 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:40:49.033452 2456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:49.034397 2456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 22:40:49 dist-test-slave-csnl krb5kdc[2354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:40:49.041889 2456 catalog_manager.cc:1347] Generated new cluster ID: 250ad77c7e164318a9ec3cc44fd6bcc2
I20260925 22:40:49.041975 2456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:49.044080 2437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:49.015227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37940 (local address 127.2.74.254:41703)
0925 22:40:49.015978 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:49.016003 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:49.017164 (+ 1161us) server_negotiation.cc:407] Connection header received
0925 22:40:49.021899 (+ 4735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:49.021930 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:49.022385 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:49.022851 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:49.025768 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:49.027413 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:49.029631 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:49.030501 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:49.030709 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:49.037036 (+ 6327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:49.037044 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:49.037065 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:49.037105 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:49.039537 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:49.040400 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:49.040407 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:49.040411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:49.040556 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:49.040969 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:49.040972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:49.040973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:49.041469 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:49.041703 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:49.042646 (+ 943us) server_negotiation.cc:299] Negotiation successful
0925 22:40:49.043026 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":157,"threads_started":1}
I20260925 22:40:49.058204 2456 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:49.059625 2456 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:49.071502 2456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58802a6c852e49efb7b1083255015618: Generated new TSK 0
I20260925 22:40:49.072446 2456 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
I20260925 22:40:49.110630 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:0
--local_ip_for_outbound_sockets=127.2.74.193
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:41703
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:33017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:49.238409 2461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:49.238719 2461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:49.238766 2461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:49.243438 2461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:49.243518 2461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:49.243584 2461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:40:49.249189 2461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:33017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:41703
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:49.250334 2461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:49.251456 2461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:49.261736 2467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:49.262006 2461 server_base.cc:1034] running on GCE node
W20260925 22:40:49.261770 2466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:49.261745 2469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:49.262583 2461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:49.263329 2461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:49.264542 2461 hybrid_clock.cc:648] HybridClock initialized: now 1790376049264521 us; error 38 us; skew 500 ppm
Sep 25 22:40:49 dist-test-slave-csnl krb5kdc[2354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376049, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:49.268543 2461 init.cc:377] Logged in from keytab as kudu/127.2.74.193@KRBTEST.COM (short username kudu)
I20260925 22:40:49.269548 2461 webserver.cc:460] Webserver started at http://127.2.74.193:34893/ using document root <none> and password file <none>
I20260925 22:40:49.270187 2461 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:49.270284 2461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:49.270539 2461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:49.272946 2461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-0/data/instance:
uuid: "7d7cf7834b914669846f91197de2db40"
format_stamp: "Formatted at 2026-09-25 22:40:49 on dist-test-slave-csnl"
server_key: "118715259a3f7bc79418c1e5173c578c"
server_key_iv: "a507ff35b7738eb27b57d226d60efb06"
server_key_version: "encryptionkey@0"
I20260925 22:40:49.273634 2461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance:
uuid: "7d7cf7834b914669846f91197de2db40"
format_stamp: "Formatted at 2026-09-25 22:40:49 on dist-test-slave-csnl"
server_key: "118715259a3f7bc79418c1e5173c578c"
server_key_iv: "a507ff35b7738eb27b57d226d60efb06"
server_key_version: "encryptionkey@0"
I20260925 22:40:49.278199 2461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:40:49.281148 2476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:49.282593 2461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:40:49.282764 2461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "7d7cf7834b914669846f91197de2db40"
format_stamp: "Formatted at 2026-09-25 22:40:49 on dist-test-slave-csnl"
server_key: "118715259a3f7bc79418c1e5173c578c"
server_key_iv: "a507ff35b7738eb27b57d226d60efb06"
server_key_version: "encryptionkey@0"
I20260925 22:40:49.282900 2461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:49.297385 2461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:49.300699 2461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:49.300959 2461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:49.301627 2461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:40:49.302558 2461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:49.302634 2461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:49.302704 2461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:49.302745 2461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:49.312999 2461 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:38371
I20260925 22:40:49.313169 2589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:38371 every 8 connection(s)
I20260925 22:40:49.314002 2461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:40:49.318312 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 2461
I20260925 22:40:49.318447 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance
I20260925 22:40:49.318706 2347 external_mini_cluster.cc:1428] Setting key 3bad3f0fb01551edbe32ebcf3d167da6
Sep 25 22:40:49 dist-test-slave-csnl krb5kdc[2354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376049, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.74.194@KRBTEST.COM; defaulting to no policy
I20260925 22:40:49.329244 2437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:49.315948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:48127 (local address 127.2.74.254:41703)
0925 22:40:49.316121 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:49.316125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:49.316985 (+ 860us) server_negotiation.cc:407] Connection header received
0925 22:40:49.318374 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:49.318377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:49.318436 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:49.318516 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:49.319944 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:49.320564 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:49.321918 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:49.322321 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:49.322443 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:49.325081 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:49.325089 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:49.325096 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:49.325133 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:49.326846 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:49.327479 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:49.327486 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:49.327490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:49.327579 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:49.327976 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:49.327980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:49.327983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:49.328257 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:49.328377 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:49.328937 (+ 560us) server_negotiation.cc:299] Negotiation successful
0925 22:40:49.329076 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 22:40:49.330230 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:49.316352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41703 (local address 127.2.74.193:48127)
0925 22:40:49.316828 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:49.316871 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:49.318163 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:49.318723 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:49.318731 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:49.319261 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:49.319801 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:49.319816 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:49.320695 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:49.320698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:49.321790 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:49.321801 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:49.322508 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:49.322511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:49.322681 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:49.323378 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:49.323397 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:49.324857 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:49.327066 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:49.327073 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:49.327087 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:49.327366 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:49.327705 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:49.327707 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:49.327709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:49.327845 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:49.328365 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:49.328370 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:49.328628 (+ 258us) client_negotiation.cc:769] Sending connection context
0925 22:40:49.328876 (+ 248us) client_negotiation.cc:240] Negotiation successful
0925 22:40:49.329248 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.194@KRBTEST.COM" created.
I20260925 22:40:49.331683 2590 heartbeater.cc:347] Connected to a master server at 127.2.74.254:41703
I20260925 22:40:49.331971 2590 heartbeater.cc:464] Registering TS with master...
I20260925 22:40:49.332638 2590 heartbeater.cc:511] Master 127.2.74.254:41703 requested a full tablet report, sending...
I20260925 22:40:49.334486 2399 ts_manager.cc:194] Registered new tserver with Master: 7d7cf7834b914669846f91197de2db40 (127.2.74.193:38371)
I20260925 22:40:49.336236 2399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.193@KRBTEST.COM'} at 127.2.74.193:48127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
I20260925 22:40:49.349426 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:0
--local_ip_for_outbound_sockets=127.2.74.194
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:41703
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:33017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:49.468871 2595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:49.469209 2595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:49.469281 2595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:49.473575 2595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:49.473676 2595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:49.473767 2595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:40:49.479425 2595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:33017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:41703
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:49.480624 2595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:49.481710 2595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:40:49.491777 2595 server_base.cc:1034] running on GCE node
W20260925 22:40:49.491716 2600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:49.491729 2603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:49.491720 2601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:49.492368 2595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:49.493119 2595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:49.494302 2595 hybrid_clock.cc:648] HybridClock initialized: now 1790376049494269 us; error 50 us; skew 500 ppm
Sep 25 22:40:49 dist-test-slave-csnl krb5kdc[2354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376049, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:49.497181 2595 init.cc:377] Logged in from keytab as kudu/127.2.74.194@KRBTEST.COM (short username kudu)
I20260925 22:40:49.498051 2595 webserver.cc:460] Webserver started at http://127.2.74.194:43587/ using document root <none> and password file <none>
I20260925 22:40:49.498600 2595 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:49.498661 2595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:49.498827 2595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:49.501044 2595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-1/data/instance:
uuid: "e816a534859245d3acda5285f5ba86dd"
format_stamp: "Formatted at 2026-09-25 22:40:49 on dist-test-slave-csnl"
server_key: "16e7efd3e9ab1028245e8d3ddef4280d"
server_key_iv: "e36398d0709614c0ca0a66348bb5ccad"
server_key_version: "encryptionkey@0"
I20260925 22:40:49.501657 2595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance:
uuid: "e816a534859245d3acda5285f5ba86dd"
format_stamp: "Formatted at 2026-09-25 22:40:49 on dist-test-slave-csnl"
server_key: "16e7efd3e9ab1028245e8d3ddef4280d"
server_key_iv: "e36398d0709614c0ca0a66348bb5ccad"
server_key_version: "encryptionkey@0"
I20260925 22:40:49.505891 2595 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:40:49.508673 2610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:49.510079 2595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:40:49.510242 2595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "e816a534859245d3acda5285f5ba86dd"
format_stamp: "Formatted at 2026-09-25 22:40:49 on dist-test-slave-csnl"
server_key: "16e7efd3e9ab1028245e8d3ddef4280d"
server_key_iv: "e36398d0709614c0ca0a66348bb5ccad"
server_key_version: "encryptionkey@0"
I20260925 22:40:49.510375 2595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:49.537238 2595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:49.540807 2595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:49.541038 2595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:49.541676 2595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:40:49.542555 2595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:49.542604 2595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:49.542675 2595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:49.542712 2595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:49.552245 2595 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:37451
I20260925 22:40:49.552268 2723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:37451 every 8 connection(s)
I20260925 22:40:49.553153 2595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:40:49.556583 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 2595
I20260925 22:40:49.556704 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance
I20260925 22:40:49.556931 2347 external_mini_cluster.cc:1428] Setting key 3ccdc5f9c3813a020e74a717f4de0227
Sep 25 22:40:49 dist-test-slave-csnl krb5kdc[2354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376049, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.74.195@KRBTEST.COM; defaulting to no policy
I20260925 22:40:49.568172 2437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:49.554989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:38733 (local address 127.2.74.254:41703)
0925 22:40:49.555150 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:49.555155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:49.556064 (+ 909us) server_negotiation.cc:407] Connection header received
0925 22:40:49.557517 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:49.557521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:49.557576 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:49.557679 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:49.559249 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:49.559766 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:49.561048 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:49.561303 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:49.561387 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:49.564298 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:49.564301 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:49.564303 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:49.564328 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:49.566039 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:49.566581 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:49.566586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:49.566588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:49.566689 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:49.567071 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:49.567074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:49.567075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:49.567262 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:49.567340 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:49.567938 (+ 598us) server_negotiation.cc:299] Negotiation successful
0925 22:40:49.568048 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.195@KRBTEST.COM" created.
I20260925 22:40:49.569221 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:49.555429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41703 (local address 127.2.74.194:38733)
0925 22:40:49.555912 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:49.555957 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:49.557253 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:49.557872 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:49.557882 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:49.558349 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:49.559094 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:49.559121 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:49.559900 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:49.559906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:49.560905 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:49.560916 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:49.561466 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:49.561473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:49.561717 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:49.562379 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:49.562406 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:49.564112 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:49.566167 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:49.566172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:49.566188 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:49.566455 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:49.566777 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:49.566780 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:49.566781 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:49.566919 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:49.567348 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:49.567353 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:49.567688 (+ 335us) client_negotiation.cc:769] Sending connection context
0925 22:40:49.567857 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 22:40:49.568142 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":111,"threads_started":1}
I20260925 22:40:49.570502 2724 heartbeater.cc:347] Connected to a master server at 127.2.74.254:41703
I20260925 22:40:49.570802 2724 heartbeater.cc:464] Registering TS with master...
I20260925 22:40:49.571450 2724 heartbeater.cc:511] Master 127.2.74.254:41703 requested a full tablet report, sending...
I20260925 22:40:49.572734 2399 ts_manager.cc:194] Registered new tserver with Master: e816a534859245d3acda5285f5ba86dd (127.2.74.194:37451)
I20260925 22:40:49.573457 2399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.194@KRBTEST.COM'} at 127.2.74.194:38733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
I20260925 22:40:49.586318 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:0
--local_ip_for_outbound_sockets=127.2.74.195
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:41703
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:33017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:49.705423 2729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:49.705719 2729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:49.705765 2729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:49.710654 2729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:49.710750 2729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:49.710850 2729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:40:49.716771 2729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:33017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:41703
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:49.717944 2729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:49.719089 2729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:49.729537 2737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:49.729521 2735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:49.729506 2734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:49.730242 2729 server_base.cc:1034] running on GCE node
I20260925 22:40:49.730708 2729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:49.731474 2729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:49.732678 2729 hybrid_clock.cc:648] HybridClock initialized: now 1790376049732650 us; error 41 us; skew 500 ppm
Sep 25 22:40:49 dist-test-slave-csnl krb5kdc[2354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376049, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:49.735677 2729 init.cc:377] Logged in from keytab as kudu/127.2.74.195@KRBTEST.COM (short username kudu)
I20260925 22:40:49.736948 2729 webserver.cc:460] Webserver started at http://127.2.74.195:35251/ using document root <none> and password file <none>
I20260925 22:40:49.737581 2729 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:49.737676 2729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:49.737900 2729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:49.740180 2729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-2/data/instance:
uuid: "a9d0ec7e53504d15b2b9bbf6ebee0b29"
format_stamp: "Formatted at 2026-09-25 22:40:49 on dist-test-slave-csnl"
server_key: "cc6557e3b668481094250c64720863bd"
server_key_iv: "a90e358dd957ea98aa82fce6aa85c4a2"
server_key_version: "encryptionkey@0"
I20260925 22:40:49.740779 2729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance:
uuid: "a9d0ec7e53504d15b2b9bbf6ebee0b29"
format_stamp: "Formatted at 2026-09-25 22:40:49 on dist-test-slave-csnl"
server_key: "cc6557e3b668481094250c64720863bd"
server_key_iv: "a90e358dd957ea98aa82fce6aa85c4a2"
server_key_version: "encryptionkey@0"
I20260925 22:40:49.745144 2729 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:40:49.747860 2744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:49.749176 2729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:40:49.749332 2729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "a9d0ec7e53504d15b2b9bbf6ebee0b29"
format_stamp: "Formatted at 2026-09-25 22:40:49 on dist-test-slave-csnl"
server_key: "cc6557e3b668481094250c64720863bd"
server_key_iv: "a90e358dd957ea98aa82fce6aa85c4a2"
server_key_version: "encryptionkey@0"
I20260925 22:40:49.749464 2729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:49.761869 2729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:49.766371 2729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:49.766628 2729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:49.767388 2729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:40:49.768301 2729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:49.768378 2729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:49.768453 2729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:49.768492 2729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:49.778391 2729 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:36323
I20260925 22:40:49.778409 2857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:36323 every 8 connection(s)
I20260925 22:40:49.779522 2729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:40:49.783358 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 2729
I20260925 22:40:49.783499 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance
I20260925 22:40:49.783761 2347 external_mini_cluster.cc:1428] Setting key e64f7dc99c42623abe0f264e58224997
Sep 25 22:40:49 dist-test-slave-csnl krb5kdc[2354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376049, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:40:49.794569 2437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:49.781476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:60895 (local address 127.2.74.254:41703)
0925 22:40:49.781620 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:49.781625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:49.782592 (+ 967us) server_negotiation.cc:407] Connection header received
0925 22:40:49.784117 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:49.784121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:49.784183 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:49.784291 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:49.785963 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:49.786661 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:49.788020 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:49.788252 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:49.788336 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:49.790915 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:49.790918 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:49.790920 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:49.790990 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:49.792611 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:49.793170 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:49.793173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:49.793174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:49.793249 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:49.793549 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:49.793551 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:49.793553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:49.793741 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:49.793825 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:49.794333 (+ 508us) server_negotiation.cc:299] Negotiation successful
0925 22:40:49.794440 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:40:49.795536 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:49.781941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41703 (local address 127.2.74.195:60895)
0925 22:40:49.782427 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:49.782476 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:49.783845 (+ 1369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:49.784517 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:49.784525 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:49.785079 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:49.785800 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:49.785822 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:49.786782 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:49.786785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:49.787910 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:49.787921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:49.788387 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:49.788389 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:49.788538 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:49.789199 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:49.789217 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:49.790714 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:49.792749 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:49.792755 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:49.792771 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:49.793075 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:49.793326 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:49.793328 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:49.793330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:49.793471 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:49.793833 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:49.793839 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:49.794117 (+ 278us) client_negotiation.cc:769] Sending connection context
0925 22:40:49.794283 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 22:40:49.794555 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":116,"threads_started":1}
I20260925 22:40:49.796761 2858 heartbeater.cc:347] Connected to a master server at 127.2.74.254:41703
I20260925 22:40:49.797050 2858 heartbeater.cc:464] Registering TS with master...
I20260925 22:40:49.797703 2858 heartbeater.cc:511] Master 127.2.74.254:41703 requested a full tablet report, sending...
I20260925 22:40:49.798799 2399 ts_manager.cc:194] Registered new tserver with Master: a9d0ec7e53504d15b2b9bbf6ebee0b29 (127.2.74.195:36323)
I20260925 22:40:49.799418 2399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.195@KRBTEST.COM'} at 127.2.74.195:60895
I20260925 22:40:49.810442 2347 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 22:40:49 dist-test-slave-csnl krb5kdc[2354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376049, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:40:49 dist-test-slave-csnl krb5kdc[2354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376049, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.74.193@KRBTEST.COM
I20260925 22:40:49.837414 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:49.827427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51688 (local address 127.2.74.193:38371)
0925 22:40:49.827721 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:49.827735 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:49.827760 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:40:49.827952 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:49.827963 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:49.828203 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:49.828326 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:49.829024 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:49.829825 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:49.830664 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:49.831167 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:49.831258 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:49.833860 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:49.833872 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:49.833892 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:49.833923 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:49.835558 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:49.836048 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:49.836054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:49.836055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:49.836147 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:49.836404 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:49.836406 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:49.836407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:49.836663 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:49.836818 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:49.837023 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 22:40:49.837201 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":89,"threads_started":1}
W20260925 22:40:49.838488 2504 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51688
Sep 25 22:40:49 dist-test-slave-csnl krb5kdc[2354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:40:49 dist-test-slave-csnl krb5kdc[2354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.193@KRBTEST.COM
I20260925 22:40:49.868995 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:49.857011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51692 (local address 127.2.74.193:38371)
0925 22:40:49.857148 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:49.857152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:49.857382 (+ 230us) server_negotiation.cc:407] Connection header received
0925 22:40:49.857465 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:49.857474 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:49.857520 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:49.857603 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:49.858334 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:49.858919 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:49.859762 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:49.859992 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:49.860072 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:49.866049 (+ 5977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:49.866052 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:49.866055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:49.866088 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:49.867525 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:49.867964 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:49.867966 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:49.867967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:49.868041 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:49.868280 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:49.868282 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:49.868284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:49.868453 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:49.868564 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:49.868763 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 22:40:49.868870 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:40:49.870188 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 2461
I20260925 22:40:49.876446 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 2595
I20260925 22:40:49.883949 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 2729
I20260925 22:40:49.889708 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 2367
2026-09-25T22:40:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2300 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:40:49 dist-test-slave-csnl krb5kdc[2872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:40:49 dist-test-slave-csnl krb5kdc[2872](info): set up 2 sockets
Sep 25 22:40:49 dist-test-slave-csnl krb5kdc[2872](info): commencing operation
krb5kdc: starting...
W20260925 22:40:50.946626 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:40:51 dist-test-slave-csnl krb5kdc[2872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:40:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:40:51.078224 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:41177
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:35101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:41177
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:51.193250 2885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:51.193576 2885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:51.193619 2885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:51.198087 2885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:51.198182 2885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:51.198203 2885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:51.198221 2885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:51.198237 2885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:51.204138 2885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:35101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:41177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:41177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:51.205375 2885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:51.206521 2885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:51.215710 2893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:51.215785 2885 server_base.cc:1034] running on GCE node
W20260925 22:40:51.215728 2891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:51.215744 2890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:51.216645 2885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:51.217936 2885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:51.219113 2885 hybrid_clock.cc:648] HybridClock initialized: now 1790376051219093 us; error 40 us; skew 500 ppm
Sep 25 22:40:51 dist-test-slave-csnl krb5kdc[2872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376051, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:51.222059 2885 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:40:51.223150 2885 webserver.cc:460] Webserver started at http://127.2.74.254:42937/ using document root <none> and password file <none>
I20260925 22:40:51.223754 2885 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:51.223848 2885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:51.224092 2885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:51.226281 2885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "d27e88cc43914aa98f306ce1bf35afba"
format_stamp: "Formatted at 2026-09-25 22:40:51 on dist-test-slave-csnl"
server_key: "8ca154006a76b0dff1dac3c927e7efe8"
server_key_iv: "4272fe46dbbdad571e23eacd14c7785e"
server_key_version: "encryptionkey@0"
I20260925 22:40:51.226893 2885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "d27e88cc43914aa98f306ce1bf35afba"
format_stamp: "Formatted at 2026-09-25 22:40:51 on dist-test-slave-csnl"
server_key: "8ca154006a76b0dff1dac3c927e7efe8"
server_key_iv: "4272fe46dbbdad571e23eacd14c7785e"
server_key_version: "encryptionkey@0"
I20260925 22:40:51.231101 2885 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:40:51.233978 2900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:51.235486 2885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:40:51.235641 2885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "d27e88cc43914aa98f306ce1bf35afba"
format_stamp: "Formatted at 2026-09-25 22:40:51 on dist-test-slave-csnl"
server_key: "8ca154006a76b0dff1dac3c927e7efe8"
server_key_iv: "4272fe46dbbdad571e23eacd14c7785e"
server_key_version: "encryptionkey@0"
I20260925 22:40:51.235772 2885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:51.267346 2885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:51.270978 2885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:51.271302 2885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:51.278789 2885 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:41177
I20260925 22:40:51.278807 2952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:41177 every 8 connection(s)
I20260925 22:40:51.279850 2885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:51.282639 2953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:51.285650 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 2885
I20260925 22:40:51.285781 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:40:51.286060 2347 external_mini_cluster.cc:1428] Setting key a68b7e2a405c9af5dbf0e9e30dcdc5c2
I20260925 22:40:51.289389 2953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d27e88cc43914aa98f306ce1bf35afba: Bootstrap starting.
I20260925 22:40:51.291855 2953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d27e88cc43914aa98f306ce1bf35afba: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:51.292667 2953 log.cc:824] T 00000000000000000000000000000000 P d27e88cc43914aa98f306ce1bf35afba: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:51.294881 2953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d27e88cc43914aa98f306ce1bf35afba: No bootstrap required, opened a new log
Sep 25 22:40:51 dist-test-slave-csnl krb5kdc[2872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:40:51.298559 2953 raft_consensus.cc:348] T 00000000000000000000000000000000 P d27e88cc43914aa98f306ce1bf35afba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d27e88cc43914aa98f306ce1bf35afba" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 41177 } }
I20260925 22:40:51.298869 2953 raft_consensus.cc:374] T 00000000000000000000000000000000 P d27e88cc43914aa98f306ce1bf35afba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:51.298954 2953 raft_consensus.cc:729] T 00000000000000000000000000000000 P d27e88cc43914aa98f306ce1bf35afba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d27e88cc43914aa98f306ce1bf35afba, State: Initialized, Role: FOLLOWER
I20260925 22:40:51.299587 2953 consensus_queue.cc:260] T 00000000000000000000000000000000 P d27e88cc43914aa98f306ce1bf35afba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d27e88cc43914aa98f306ce1bf35afba" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 41177 } }
I20260925 22:40:51.299798 2958 raft_consensus.cc:482] T 00000000000000000000000000000000 P d27e88cc43914aa98f306ce1bf35afba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:40:51.299973 2958 raft_consensus.cc:504] T 00000000000000000000000000000000 P d27e88cc43914aa98f306ce1bf35afba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d27e88cc43914aa98f306ce1bf35afba" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 41177 } }
I20260925 22:40:51.299773 2953 raft_consensus.cc:388] T 00000000000000000000000000000000 P d27e88cc43914aa98f306ce1bf35afba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:40:51.300421 2953 raft_consensus.cc:482] T 00000000000000000000000000000000 P d27e88cc43914aa98f306ce1bf35afba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:40:51.300387 2958 leader_election.cc:302] T 00000000000000000000000000000000 P d27e88cc43914aa98f306ce1bf35afba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d27e88cc43914aa98f306ce1bf35afba; no voters:
I20260925 22:40:51.300544 2953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d27e88cc43914aa98f306ce1bf35afba [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:51.300968 2958 leader_election.cc:288] T 00000000000000000000000000000000 P d27e88cc43914aa98f306ce1bf35afba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260925 22:40:51.301695 2956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:51.287734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48546 (local address 127.2.74.254:41177)
0925 22:40:51.288246 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:51.288262 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:51.288301 (+ 39us) server_negotiation.cc:407] Connection header received
0925 22:40:51.288944 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:51.288962 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:51.289330 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:51.289709 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:51.290535 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:51.291677 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:51.292630 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:51.293174 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:51.293347 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:51.296058 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:51.296064 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:51.296080 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:51.296112 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:51.298309 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:51.298841 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:51.298852 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:51.298859 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:51.299062 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:51.299370 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:51.299374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:51.299376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:51.299879 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:51.300124 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:51.300399 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 22:40:51.300760 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":149,"threads_started":1}
I20260925 22:40:51.301774 2953 raft_consensus.cc:504] T 00000000000000000000000000000000 P d27e88cc43914aa98f306ce1bf35afba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d27e88cc43914aa98f306ce1bf35afba" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 41177 } }
I20260925 22:40:51.301960 2953 leader_election.cc:302] T 00000000000000000000000000000000 P d27e88cc43914aa98f306ce1bf35afba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d27e88cc43914aa98f306ce1bf35afba; no voters:
I20260925 22:40:51.301998 2959 raft_consensus.cc:2741] T 00000000000000000000000000000000 P d27e88cc43914aa98f306ce1bf35afba [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260925 22:40:51.302070 2953 leader_election.cc:288] T 00000000000000000000000000000000 P d27e88cc43914aa98f306ce1bf35afba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:40:51.302407 2960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d27e88cc43914aa98f306ce1bf35afba [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:51.302671 2960 raft_consensus.cc:686] T 00000000000000000000000000000000 P d27e88cc43914aa98f306ce1bf35afba [term 1 LEADER]: Becoming Leader. State: Replica: d27e88cc43914aa98f306ce1bf35afba, State: Running, Role: LEADER
I20260925 22:40:51.303092 2960 consensus_queue.cc:237] T 00000000000000000000000000000000 P d27e88cc43914aa98f306ce1bf35afba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d27e88cc43914aa98f306ce1bf35afba" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 41177 } }
I20260925 22:40:51.304415 2959 sys_catalog.cc:455] T 00000000000000000000000000000000 P d27e88cc43914aa98f306ce1bf35afba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d27e88cc43914aa98f306ce1bf35afba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d27e88cc43914aa98f306ce1bf35afba" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 41177 } } }
I20260925 22:40:51.304566 2959 sys_catalog.cc:458] T 00000000000000000000000000000000 P d27e88cc43914aa98f306ce1bf35afba [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:51.304673 2953 sys_catalog.cc:564] T 00000000000000000000000000000000 P d27e88cc43914aa98f306ce1bf35afba [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:51.304870 2958 sys_catalog.cc:455] T 00000000000000000000000000000000 P d27e88cc43914aa98f306ce1bf35afba [sys.catalog]: SysCatalogTable state changed. Reason: New leader d27e88cc43914aa98f306ce1bf35afba. Latest consensus state: current_term: 1 leader_uuid: "d27e88cc43914aa98f306ce1bf35afba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d27e88cc43914aa98f306ce1bf35afba" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 41177 } } }
I20260925 22:40:51.304946 2958 sys_catalog.cc:458] T 00000000000000000000000000000000 P d27e88cc43914aa98f306ce1bf35afba [sys.catalog]: This master's current role is: LEADER
W20260925 22:40:51.309579 2975 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d27e88cc43914aa98f306ce1bf35afba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:40:51.309700 2975 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:40:51.309783 2963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:51.310771 2963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:40:51.317437 2963 catalog_manager.cc:1347] Generated new cluster ID: 3b348d90de0143f7a5fc2b2b17cd8de1
I20260925 22:40:51.317535 2963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:51.334491 2963 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:51.335619 2963 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:51.345973 2963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d27e88cc43914aa98f306ce1bf35afba: Generated new TSK 0
I20260925 22:40:51.346921 2963 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
I20260925 22:40:51.381938 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:0
--local_ip_for_outbound_sockets=127.2.74.193
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:41177
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:35101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:51.501458 2980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:51.501760 2980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:51.501807 2980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:51.506258 2980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:51.506335 2980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:51.506436 2980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:40:51.512302 2980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:35101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:41177
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:51.513520 2980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:51.514591 2980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:51.524879 2988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:51.524911 2986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:51.524879 2985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:51.525310 2980 server_base.cc:1034] running on GCE node
I20260925 22:40:51.525691 2980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:51.526367 2980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:51.527580 2980 hybrid_clock.cc:648] HybridClock initialized: now 1790376051527555 us; error 50 us; skew 500 ppm
Sep 25 22:40:51 dist-test-slave-csnl krb5kdc[2872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376051, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:51.530543 2980 init.cc:377] Logged in from keytab as kudu/127.2.74.193@KRBTEST.COM (short username kudu)
I20260925 22:40:51.531603 2980 webserver.cc:460] Webserver started at http://127.2.74.193:37589/ using document root <none> and password file <none>
I20260925 22:40:51.532152 2980 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:51.532213 2980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:51.532384 2980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:51.534456 2980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-0/data/instance:
uuid: "5d11b70782f44b4bbe7135aced20a8d8"
format_stamp: "Formatted at 2026-09-25 22:40:51 on dist-test-slave-csnl"
server_key: "df0fabff2e04e68be0e8d3d036f2dcf4"
server_key_iv: "b69896d0944790ba2926d71a84d1a1bc"
server_key_version: "encryptionkey@0"
I20260925 22:40:51.535079 2980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance:
uuid: "5d11b70782f44b4bbe7135aced20a8d8"
format_stamp: "Formatted at 2026-09-25 22:40:51 on dist-test-slave-csnl"
server_key: "df0fabff2e04e68be0e8d3d036f2dcf4"
server_key_iv: "b69896d0944790ba2926d71a84d1a1bc"
server_key_version: "encryptionkey@0"
I20260925 22:40:51.539095 2980 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:40:51.541749 2995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:51.543093 2980 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:40:51.543233 2980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "5d11b70782f44b4bbe7135aced20a8d8"
format_stamp: "Formatted at 2026-09-25 22:40:51 on dist-test-slave-csnl"
server_key: "df0fabff2e04e68be0e8d3d036f2dcf4"
server_key_iv: "b69896d0944790ba2926d71a84d1a1bc"
server_key_version: "encryptionkey@0"
I20260925 22:40:51.543354 2980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:51.554602 2980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:51.557647 2980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:51.557883 2980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:51.558503 2980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:40:51.559469 2980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:51.559518 2980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:51.559557 2980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:51.559583 2980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:51.569085 2980 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:43327
I20260925 22:40:51.569101 3108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:43327 every 8 connection(s)
I20260925 22:40:51.570047 2980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:40:51.579192 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 2980
I20260925 22:40:51.579334 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance
I20260925 22:40:51.579582 2347 external_mini_cluster.cc:1428] Setting key f52581d5042ecca1cac2f9fa1cd8f6de
Sep 25 22:40:51 dist-test-slave-csnl krb5kdc[2872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376051, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:40:51.585089 2956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:51.572013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:38701 (local address 127.2.74.254:41177)
0925 22:40:51.572169 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:51.572173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:51.573043 (+ 870us) server_negotiation.cc:407] Connection header received
0925 22:40:51.574320 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:51.574323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:51.574373 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:51.574458 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:51.575742 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:51.576266 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:51.577426 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:51.577646 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:51.577722 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:51.580522 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:51.580531 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:51.580534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:51.580559 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:51.582397 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:51.583194 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:51.583197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:51.583198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:51.583284 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:51.583787 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:51.583790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:51.583791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:51.584008 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:51.584091 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:51.584821 (+ 730us) server_negotiation.cc:299] Negotiation successful
0925 22:40:51.584940 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:40:51.586525 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:51.572374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41177 (local address 127.2.74.193:38701)
0925 22:40:51.572844 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:51.572889 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:51.574117 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:51.574626 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:51.574633 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:51.575101 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:51.575603 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:51.575617 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:51.576376 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:51.576378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:51.577305 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:51.577315 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:51.577775 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:51.577777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:51.577938 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:51.578519 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:51.578536 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:51.580338 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:51.582534 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:51.582540 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:51.582555 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:51.583056 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:51.583406 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:51.583412 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:51.583415 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:51.583645 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:51.584120 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:51.584128 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:51.584539 (+ 411us) client_negotiation.cc:769] Sending connection context
0925 22:40:51.584778 (+ 239us) client_negotiation.cc:240] Negotiation successful
0925 22:40:51.585200 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":130,"threads_started":1}
I20260925 22:40:51.588106 3109 heartbeater.cc:347] Connected to a master server at 127.2.74.254:41177
I20260925 22:40:51.588366 3109 heartbeater.cc:464] Registering TS with master...
I20260925 22:40:51.588886 3109 heartbeater.cc:511] Master 127.2.74.254:41177 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.74.194@KRBTEST.COM; defaulting to no policy
I20260925 22:40:51.590550 2914 ts_manager.cc:194] Registered new tserver with Master: 5d11b70782f44b4bbe7135aced20a8d8 (127.2.74.193:43327)
I20260925 22:40:51.591764 2914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.193@KRBTEST.COM'} at 127.2.74.193:38701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
I20260925 22:40:51.607789 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:0
--local_ip_for_outbound_sockets=127.2.74.194
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:41177
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:35101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:51.722414 3114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:51.722692 3114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:51.722740 3114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:51.727018 3114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:51.727093 3114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:51.727205 3114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:40:51.732915 3114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:35101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:41177
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:51.734081 3114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:51.735221 3114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:51.745189 3120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:51.745159 3122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:51.745239 3119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:51.745460 3114 server_base.cc:1034] running on GCE node
I20260925 22:40:51.745931 3114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:51.746582 3114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:51.747730 3114 hybrid_clock.cc:648] HybridClock initialized: now 1790376051747712 us; error 32 us; skew 500 ppm
Sep 25 22:40:51 dist-test-slave-csnl krb5kdc[2872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376051, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:51.750458 3114 init.cc:377] Logged in from keytab as kudu/127.2.74.194@KRBTEST.COM (short username kudu)
I20260925 22:40:51.751430 3114 webserver.cc:460] Webserver started at http://127.2.74.194:37411/ using document root <none> and password file <none>
I20260925 22:40:51.751956 3114 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:51.752013 3114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:51.752180 3114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:51.754192 3114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-1/data/instance:
uuid: "de156a260db8484ba7ab55a3d32fcba5"
format_stamp: "Formatted at 2026-09-25 22:40:51 on dist-test-slave-csnl"
server_key: "3ea6d01b684c443716eacef2e5792e91"
server_key_iv: "e0a3e42f35b2b4a62c24c9f11f2f323a"
server_key_version: "encryptionkey@0"
I20260925 22:40:51.754768 3114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance:
uuid: "de156a260db8484ba7ab55a3d32fcba5"
format_stamp: "Formatted at 2026-09-25 22:40:51 on dist-test-slave-csnl"
server_key: "3ea6d01b684c443716eacef2e5792e91"
server_key_iv: "e0a3e42f35b2b4a62c24c9f11f2f323a"
server_key_version: "encryptionkey@0"
I20260925 22:40:51.758649 3114 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260925 22:40:51.761372 3129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:51.762686 3114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:40:51.762835 3114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "de156a260db8484ba7ab55a3d32fcba5"
format_stamp: "Formatted at 2026-09-25 22:40:51 on dist-test-slave-csnl"
server_key: "3ea6d01b684c443716eacef2e5792e91"
server_key_iv: "e0a3e42f35b2b4a62c24c9f11f2f323a"
server_key_version: "encryptionkey@0"
I20260925 22:40:51.762977 3114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:51.785609 3114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:51.789041 3114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:51.789247 3114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:51.789863 3114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:40:51.790709 3114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:51.790778 3114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:51.790848 3114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:51.790892 3114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:51.799937 3114 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:37235
I20260925 22:40:51.799959 3242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:37235 every 8 connection(s)
I20260925 22:40:51.800915 3114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:40:51.804452 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 3114
I20260925 22:40:51.804548 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance
I20260925 22:40:51.804852 2347 external_mini_cluster.cc:1428] Setting key 148cfa3142666e1d3cc0e4d8cf5304bb
Sep 25 22:40:51 dist-test-slave-csnl krb5kdc[2872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376051, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.74.195@KRBTEST.COM; defaulting to no policy
I20260925 22:40:51.816454 2956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:51.802714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:60679 (local address 127.2.74.254:41177)
0925 22:40:51.802851 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:51.802855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:51.803871 (+ 1016us) server_negotiation.cc:407] Connection header received
0925 22:40:51.805652 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:51.805656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:51.805714 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:51.805799 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:51.807413 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:51.808145 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:51.809360 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:51.809789 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:51.809906 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:51.812624 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:51.812628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:51.812631 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:51.812665 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:51.814218 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:51.814805 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:51.814809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:51.814810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:51.814892 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:51.815268 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:51.815271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:51.815274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:51.815526 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:51.815638 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:51.816148 (+ 510us) server_negotiation.cc:299] Negotiation successful
0925 22:40:51.816278 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.195@KRBTEST.COM" created.
I20260925 22:40:51.817443 3245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:51.803198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41177 (local address 127.2.74.194:60679)
0925 22:40:51.803687 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:51.803746 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:51.805370 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:51.805988 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:51.805996 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:51.806534 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:51.807255 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:51.807274 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:51.808291 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:51.808294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:51.809226 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:51.809237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:51.809963 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:51.809966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:51.810132 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:51.810803 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:51.810821 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:51.812345 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:51.814346 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:51.814352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:51.814376 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:51.814698 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:51.815018 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:51.815020 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:51.815022 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:51.815165 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:51.815633 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:51.815640 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:51.815901 (+ 261us) client_negotiation.cc:769] Sending connection context
0925 22:40:51.816090 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 22:40:51.816445 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":115,"threads_started":1}
I20260925 22:40:51.818876 3243 heartbeater.cc:347] Connected to a master server at 127.2.74.254:41177
I20260925 22:40:51.819173 3243 heartbeater.cc:464] Registering TS with master...
I20260925 22:40:51.819707 3243 heartbeater.cc:511] Master 127.2.74.254:41177 requested a full tablet report, sending...
I20260925 22:40:51.820786 2914 ts_manager.cc:194] Registered new tserver with Master: de156a260db8484ba7ab55a3d32fcba5 (127.2.74.194:37235)
I20260925 22:40:51.821412 2914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.194@KRBTEST.COM'} at 127.2.74.194:60679
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
I20260925 22:40:51.833739 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:0
--local_ip_for_outbound_sockets=127.2.74.195
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:41177
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:35101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:51.951804 3248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:51.952148 3248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:51.952232 3248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:51.956519 3248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:51.956645 3248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:51.956746 3248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:40:51.962601 3248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:35101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:41177
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:51.963856 3248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:51.964975 3248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:51.975543 3254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:51.975543 3256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:51.975611 3248 server_base.cc:1034] running on GCE node
W20260925 22:40:51.975543 3253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:51.976369 3248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:51.977048 3248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:51.978221 3248 hybrid_clock.cc:648] HybridClock initialized: now 1790376051978205 us; error 32 us; skew 500 ppm
Sep 25 22:40:51 dist-test-slave-csnl krb5kdc[2872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376051, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:51.981287 3248 init.cc:377] Logged in from keytab as kudu/127.2.74.195@KRBTEST.COM (short username kudu)
I20260925 22:40:51.982215 3248 webserver.cc:460] Webserver started at http://127.2.74.195:44847/ using document root <none> and password file <none>
I20260925 22:40:51.982774 3248 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:51.982837 3248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:51.983081 3248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:51.985229 3248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-2/data/instance:
uuid: "f84040238f8147d59e2d650ee4713639"
format_stamp: "Formatted at 2026-09-25 22:40:51 on dist-test-slave-csnl"
server_key: "b57e516800367b78588ee970488219cc"
server_key_iv: "1612d980cbd5e10f9e26b3f6612700bc"
server_key_version: "encryptionkey@0"
I20260925 22:40:51.985786 3248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance:
uuid: "f84040238f8147d59e2d650ee4713639"
format_stamp: "Formatted at 2026-09-25 22:40:51 on dist-test-slave-csnl"
server_key: "b57e516800367b78588ee970488219cc"
server_key_iv: "1612d980cbd5e10f9e26b3f6612700bc"
server_key_version: "encryptionkey@0"
I20260925 22:40:51.989845 3248 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:40:51.992480 3263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:51.993841 3248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:40:51.993994 3248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "f84040238f8147d59e2d650ee4713639"
format_stamp: "Formatted at 2026-09-25 22:40:51 on dist-test-slave-csnl"
server_key: "b57e516800367b78588ee970488219cc"
server_key_iv: "1612d980cbd5e10f9e26b3f6612700bc"
server_key_version: "encryptionkey@0"
I20260925 22:40:51.994132 3248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:52.008006 3248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:52.011520 3248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:52.011729 3248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:52.012403 3248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:40:52.013366 3248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:52.013443 3248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:52.013535 3248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:52.013581 3248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:52.025614 3248 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:44171
I20260925 22:40:52.025630 3376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:44171 every 8 connection(s)
I20260925 22:40:52.026607 3248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:40:52.030509 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 3248
I20260925 22:40:52.030620 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance
I20260925 22:40:52.030876 2347 external_mini_cluster.cc:1428] Setting key 9f547b422a1c515272a4c35a62a833e6
Sep 25 22:40:52 dist-test-slave-csnl krb5kdc[2872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376051, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:40:52.041564 2956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:52.028585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:43795 (local address 127.2.74.254:41177)
0925 22:40:52.028762 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:52.028770 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:52.029701 (+ 931us) server_negotiation.cc:407] Connection header received
0925 22:40:52.031316 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:52.031323 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:52.031395 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:52.031498 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:52.033026 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.033561 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.034776 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.035079 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.035188 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:52.037752 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:52.037755 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:52.037757 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:52.037783 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:52.039278 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:52.039909 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:52.039913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:52.039915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:52.039999 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:52.040355 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:52.040358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:52.040360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:52.040561 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:52.040667 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:52.041237 (+ 570us) server_negotiation.cc:299] Negotiation successful
0925 22:40:52.041381 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 22:40:52.042419 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:52.029034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41177 (local address 127.2.74.195:43795)
0925 22:40:52.029520 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:52.029570 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:52.031082 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:52.031738 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:52.031751 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:52.032217 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:52.032811 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:52.032849 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.033699 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.033702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:52.034636 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:52.034646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.035238 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.035242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:52.035425 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:52.035988 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:52.036023 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:52.037552 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:52.039438 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:52.039445 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:52.039469 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:52.039801 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:52.040097 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:52.040100 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:52.040102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:52.040256 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:52.040665 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:52.040673 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:52.040958 (+ 285us) client_negotiation.cc:769] Sending connection context
0925 22:40:52.041176 (+ 218us) client_negotiation.cc:240] Negotiation successful
0925 22:40:52.041535 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":126,"threads_started":1}
I20260925 22:40:52.043726 3377 heartbeater.cc:347] Connected to a master server at 127.2.74.254:41177
I20260925 22:40:52.044035 3377 heartbeater.cc:464] Registering TS with master...
I20260925 22:40:52.044687 3377 heartbeater.cc:511] Master 127.2.74.254:41177 requested a full tablet report, sending...
I20260925 22:40:52.045710 2914 ts_manager.cc:194] Registered new tserver with Master: f84040238f8147d59e2d650ee4713639 (127.2.74.195:44171)
I20260925 22:40:52.046336 2914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.195@KRBTEST.COM'} at 127.2.74.195:43795
I20260925 22:40:52.057391 2347 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:40:52.068840 2956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:52.061138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48548 (local address 127.2.74.254:41177)
0925 22:40:52.061275 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:52.061279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:52.061511 (+ 232us) server_negotiation.cc:407] Connection header received
0925 22:40:52.061637 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:52.061640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:52.061684 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:52.061761 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:52.062469 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.063047 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.063861 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.064075 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.064149 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:52.065463 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:52.065474 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:52.065476 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:52.065500 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:52.066989 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:52.067429 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:52.067435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:52.067438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:52.067539 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:52.067857 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:52.067859 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:52.067860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:52.068106 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:52.068266 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:52.068486 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 22:40:52.068635 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:40:52.074687 2914 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:40:52.078749 2914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:40:52.094447 3391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:52.088233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45172 (local address 127.2.74.193:43327)
0925 22:40:52.089323 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:52.089329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:52.089350 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:40:52.089432 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:52.089440 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:52.089658 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:52.089809 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:52.090089 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.091193 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.092943 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.093699 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.093829 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:52.093993 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:52.094169 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 22:40:52.094271 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":983,"thread_start_us":94,"threads_started":1}
I20260925 22:40:52.094434 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:52.088101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:43327 (local address 127.0.0.1:45172)
0925 22:40:52.089035 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:52.089061 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:52.089185 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:52.089808 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:52.089813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:52.089835 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:52.089965 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:52.089971 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.091288 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.091295 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:52.092784 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:52.092795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.093830 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.093833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:52.093951 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:52.093989 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 22:40:52.094133 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 22:40:52.094271 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"mutex_wait_us":103,"thread_start_us":111,"threads_started":1}
I20260925 22:40:52.094694 3388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:52.088733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:44171 (local address 127.0.0.1:55590)
0925 22:40:52.089388 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:52.089400 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:52.089489 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:52.089946 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:52.089949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:52.089967 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:52.090066 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:52.090075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.091209 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.091212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:52.092767 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:52.092777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.094063 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.094067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:52.094167 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:52.094193 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 22:40:52.094272 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 22:40:52.094355 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":179,"thread_start_us":80,"threads_started":1}
I20260925 22:40:52.094827 3393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:52.088901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55590 (local address 127.2.74.195:44171)
0925 22:40:52.089221 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:52.089231 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:52.089580 (+ 349us) server_negotiation.cc:407] Connection header received
0925 22:40:52.089667 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:52.089672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:52.089812 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:52.089965 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:52.090184 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.091076 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.092907 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.093717 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.093833 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:52.093993 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:52.094525 (+ 532us) server_negotiation.cc:299] Negotiation successful
0925 22:40:52.094648 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":149,"threads_started":1}
I20260925 22:40:52.095177 3390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:52.087927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:37235 (local address 127.0.0.1:44882)
0925 22:40:52.089000 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:52.089055 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:52.089957 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:52.090606 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:52.090608 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:52.090625 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:52.090699 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:52.090704 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.092170 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.092174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:52.093309 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:52.093319 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.094676 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.094680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:52.094788 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:52.094812 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:40:52.094894 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 22:40:52.095040 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"mutex_wait_us":3,"thread_start_us":108,"threads_started":1}
I20260925 22:40:52.095647 3389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:52.088081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44882 (local address 127.2.74.194:37235)
0925 22:40:52.088731 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:52.088743 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:52.089139 (+ 396us) server_negotiation.cc:407] Connection header received
0925 22:40:52.090127 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:52.090135 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:52.090308 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:52.090424 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:52.090820 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.092014 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.093449 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.094345 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.094460 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:52.094610 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:52.095369 (+ 759us) server_negotiation.cc:299] Negotiation successful
0925 22:40:52.095484 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":112,"threads_started":1}
I20260925 22:40:52.098201 3043 tablet_service.cc:1467] Processing CreateTablet for tablet bc4aee83e82f45c88da0f8c974ccb398 (DEFAULT_TABLE table=test-table [id=1907fd16122a4b14bfbe9c0c5618fc54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:52.098337 3311 tablet_service.cc:1467] Processing CreateTablet for tablet bc4aee83e82f45c88da0f8c974ccb398 (DEFAULT_TABLE table=test-table [id=1907fd16122a4b14bfbe9c0c5618fc54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:52.099496 3043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc4aee83e82f45c88da0f8c974ccb398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:52.099542 3311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc4aee83e82f45c88da0f8c974ccb398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:52.099088 3177 tablet_service.cc:1467] Processing CreateTablet for tablet bc4aee83e82f45c88da0f8c974ccb398 (DEFAULT_TABLE table=test-table [id=1907fd16122a4b14bfbe9c0c5618fc54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:52.100211 3177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc4aee83e82f45c88da0f8c974ccb398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:52.106374 3394 tablet_bootstrap.cc:492] T bc4aee83e82f45c88da0f8c974ccb398 P 5d11b70782f44b4bbe7135aced20a8d8: Bootstrap starting.
I20260925 22:40:52.107921 3395 tablet_bootstrap.cc:492] T bc4aee83e82f45c88da0f8c974ccb398 P f84040238f8147d59e2d650ee4713639: Bootstrap starting.
I20260925 22:40:52.108894 3396 tablet_bootstrap.cc:492] T bc4aee83e82f45c88da0f8c974ccb398 P de156a260db8484ba7ab55a3d32fcba5: Bootstrap starting.
I20260925 22:40:52.110002 3394 tablet_bootstrap.cc:654] T bc4aee83e82f45c88da0f8c974ccb398 P 5d11b70782f44b4bbe7135aced20a8d8: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:52.110042 3395 tablet_bootstrap.cc:654] T bc4aee83e82f45c88da0f8c974ccb398 P f84040238f8147d59e2d650ee4713639: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:52.111034 3394 log.cc:824] T bc4aee83e82f45c88da0f8c974ccb398 P 5d11b70782f44b4bbe7135aced20a8d8: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:52.111215 3395 log.cc:824] T bc4aee83e82f45c88da0f8c974ccb398 P f84040238f8147d59e2d650ee4713639: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:52.111953 3396 tablet_bootstrap.cc:654] T bc4aee83e82f45c88da0f8c974ccb398 P de156a260db8484ba7ab55a3d32fcba5: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:52.112735 3396 log.cc:824] T bc4aee83e82f45c88da0f8c974ccb398 P de156a260db8484ba7ab55a3d32fcba5: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:52.113237 3394 tablet_bootstrap.cc:492] T bc4aee83e82f45c88da0f8c974ccb398 P 5d11b70782f44b4bbe7135aced20a8d8: No bootstrap required, opened a new log
I20260925 22:40:52.113462 3394 ts_tablet_manager.cc:1397] T bc4aee83e82f45c88da0f8c974ccb398 P 5d11b70782f44b4bbe7135aced20a8d8: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:40:52.115029 3396 tablet_bootstrap.cc:492] T bc4aee83e82f45c88da0f8c974ccb398 P de156a260db8484ba7ab55a3d32fcba5: No bootstrap required, opened a new log
I20260925 22:40:52.115029 3395 tablet_bootstrap.cc:492] T bc4aee83e82f45c88da0f8c974ccb398 P f84040238f8147d59e2d650ee4713639: No bootstrap required, opened a new log
I20260925 22:40:52.115248 3395 ts_tablet_manager.cc:1397] T bc4aee83e82f45c88da0f8c974ccb398 P f84040238f8147d59e2d650ee4713639: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260925 22:40:52.115248 3396 ts_tablet_manager.cc:1397] T bc4aee83e82f45c88da0f8c974ccb398 P de156a260db8484ba7ab55a3d32fcba5: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:40:52.117430 3394 raft_consensus.cc:348] T bc4aee83e82f45c88da0f8c974ccb398 P 5d11b70782f44b4bbe7135aced20a8d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de156a260db8484ba7ab55a3d32fcba5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37235 } } peers { permanent_uuid: "5d11b70782f44b4bbe7135aced20a8d8" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43327 } } peers { permanent_uuid: "f84040238f8147d59e2d650ee4713639" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 44171 } }
I20260925 22:40:52.117720 3394 raft_consensus.cc:374] T bc4aee83e82f45c88da0f8c974ccb398 P 5d11b70782f44b4bbe7135aced20a8d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:52.117805 3394 raft_consensus.cc:729] T bc4aee83e82f45c88da0f8c974ccb398 P 5d11b70782f44b4bbe7135aced20a8d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d11b70782f44b4bbe7135aced20a8d8, State: Initialized, Role: FOLLOWER
I20260925 22:40:52.118263 3394 consensus_queue.cc:260] T bc4aee83e82f45c88da0f8c974ccb398 P 5d11b70782f44b4bbe7135aced20a8d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de156a260db8484ba7ab55a3d32fcba5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37235 } } peers { permanent_uuid: "5d11b70782f44b4bbe7135aced20a8d8" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43327 } } peers { permanent_uuid: "f84040238f8147d59e2d650ee4713639" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 44171 } }
I20260925 22:40:52.119051 3109 heartbeater.cc:503] Master 127.2.74.254:41177 was elected leader, sending a full tablet report...
I20260925 22:40:52.119469 3394 ts_tablet_manager.cc:1428] T bc4aee83e82f45c88da0f8c974ccb398 P 5d11b70782f44b4bbe7135aced20a8d8: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:40:52.120170 3395 raft_consensus.cc:348] T bc4aee83e82f45c88da0f8c974ccb398 P f84040238f8147d59e2d650ee4713639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de156a260db8484ba7ab55a3d32fcba5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37235 } } peers { permanent_uuid: "5d11b70782f44b4bbe7135aced20a8d8" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43327 } } peers { permanent_uuid: "f84040238f8147d59e2d650ee4713639" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 44171 } }
I20260925 22:40:52.120170 3396 raft_consensus.cc:348] T bc4aee83e82f45c88da0f8c974ccb398 P de156a260db8484ba7ab55a3d32fcba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de156a260db8484ba7ab55a3d32fcba5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37235 } } peers { permanent_uuid: "5d11b70782f44b4bbe7135aced20a8d8" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43327 } } peers { permanent_uuid: "f84040238f8147d59e2d650ee4713639" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 44171 } }
I20260925 22:40:52.120518 3396 raft_consensus.cc:374] T bc4aee83e82f45c88da0f8c974ccb398 P de156a260db8484ba7ab55a3d32fcba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:52.120527 3395 raft_consensus.cc:374] T bc4aee83e82f45c88da0f8c974ccb398 P f84040238f8147d59e2d650ee4713639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:52.120589 3395 raft_consensus.cc:729] T bc4aee83e82f45c88da0f8c974ccb398 P f84040238f8147d59e2d650ee4713639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f84040238f8147d59e2d650ee4713639, State: Initialized, Role: FOLLOWER
I20260925 22:40:52.120637 3396 raft_consensus.cc:729] T bc4aee83e82f45c88da0f8c974ccb398 P de156a260db8484ba7ab55a3d32fcba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de156a260db8484ba7ab55a3d32fcba5, State: Initialized, Role: FOLLOWER
I20260925 22:40:52.121182 3395 consensus_queue.cc:260] T bc4aee83e82f45c88da0f8c974ccb398 P f84040238f8147d59e2d650ee4713639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de156a260db8484ba7ab55a3d32fcba5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37235 } } peers { permanent_uuid: "5d11b70782f44b4bbe7135aced20a8d8" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43327 } } peers { permanent_uuid: "f84040238f8147d59e2d650ee4713639" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 44171 } }
I20260925 22:40:52.121183 3396 consensus_queue.cc:260] T bc4aee83e82f45c88da0f8c974ccb398 P de156a260db8484ba7ab55a3d32fcba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de156a260db8484ba7ab55a3d32fcba5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37235 } } peers { permanent_uuid: "5d11b70782f44b4bbe7135aced20a8d8" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43327 } } peers { permanent_uuid: "f84040238f8147d59e2d650ee4713639" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 44171 } }
I20260925 22:40:52.122408 3395 ts_tablet_manager.cc:1428] T bc4aee83e82f45c88da0f8c974ccb398 P f84040238f8147d59e2d650ee4713639: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:40:52.122408 3396 ts_tablet_manager.cc:1428] T bc4aee83e82f45c88da0f8c974ccb398 P de156a260db8484ba7ab55a3d32fcba5: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:40:52.122714 3377 heartbeater.cc:503] Master 127.2.74.254:41177 was elected leader, sending a full tablet report...
I20260925 22:40:52.122825 3243 heartbeater.cc:503] Master 127.2.74.254:41177 was elected leader, sending a full tablet report...
I20260925 22:40:52.128361 3402 raft_consensus.cc:482] T bc4aee83e82f45c88da0f8c974ccb398 P f84040238f8147d59e2d650ee4713639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:40:52.128603 3402 raft_consensus.cc:504] T bc4aee83e82f45c88da0f8c974ccb398 P f84040238f8147d59e2d650ee4713639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de156a260db8484ba7ab55a3d32fcba5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37235 } } peers { permanent_uuid: "5d11b70782f44b4bbe7135aced20a8d8" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43327 } } peers { permanent_uuid: "f84040238f8147d59e2d650ee4713639" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 44171 } }
I20260925 22:40:52.129798 3402 leader_election.cc:288] T bc4aee83e82f45c88da0f8c974ccb398 P f84040238f8147d59e2d650ee4713639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de156a260db8484ba7ab55a3d32fcba5 (127.2.74.194:37235), 5d11b70782f44b4bbe7135aced20a8d8 (127.2.74.193:43327)
I20260925 22:40:52.134119 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:52.129991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:37235 (local address 127.2.74.195:41299)
0925 22:40:52.130304 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:52.130323 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:52.130414 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:52.130868 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:52.130872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:52.130887 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:52.131157 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:52.131170 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.132018 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.132021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:52.132847 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:52.132856 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.133610 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.133617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:52.133778 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:52.133801 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:40:52.133862 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 22:40:52.133926 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":51}
I20260925 22:40:52.134140 3389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:52.130378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:41299 (local address 127.2.74.194:37235)
0925 22:40:52.130523 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:52.130529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:52.130546 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:40:52.130603 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:52.130606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:52.130674 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:52.130760 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:52.131319 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.131877 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.132973 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.133489 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.133595 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:52.133678 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:52.133902 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 22:40:52.133997 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:40:52.134200 3391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:52.130392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:53697 (local address 127.2.74.193:43327)
0925 22:40:52.130517 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:52.130522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:52.130721 (+ 199us) server_negotiation.cc:407] Connection header received
0925 22:40:52.130831 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:52.130835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:52.130884 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:52.130992 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:52.131335 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.131850 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.133132 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.133559 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.133677 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:52.133739 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:52.133997 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 22:40:52.134073 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:40:52.134204 3403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:52.130104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:43327 (local address 127.2.74.195:53697)
0925 22:40:52.130611 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:52.130625 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:52.130738 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:52.131026 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:52.131029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:52.131044 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:52.131212 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:52.131224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.131975 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.131982 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:52.133005 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:52.133016 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.133685 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.133692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:52.133838 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:52.133890 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 22:40:52.133972 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 22:40:52.134045 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":91,"thread_start_us":132,"threads_started":1}
I20260925 22:40:52.134779 3063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc4aee83e82f45c88da0f8c974ccb398" candidate_uuid: "f84040238f8147d59e2d650ee4713639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d11b70782f44b4bbe7135aced20a8d8" is_pre_election: true
I20260925 22:40:52.135126 3063 raft_consensus.cc:2454] T bc4aee83e82f45c88da0f8c974ccb398 P 5d11b70782f44b4bbe7135aced20a8d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f84040238f8147d59e2d650ee4713639 in term 0.
I20260925 22:40:52.135187 3197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc4aee83e82f45c88da0f8c974ccb398" candidate_uuid: "f84040238f8147d59e2d650ee4713639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de156a260db8484ba7ab55a3d32fcba5" is_pre_election: true
I20260925 22:40:52.135519 3197 raft_consensus.cc:2454] T bc4aee83e82f45c88da0f8c974ccb398 P de156a260db8484ba7ab55a3d32fcba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f84040238f8147d59e2d650ee4713639 in term 0.
I20260925 22:40:52.135684 3265 leader_election.cc:302] T bc4aee83e82f45c88da0f8c974ccb398 P f84040238f8147d59e2d650ee4713639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d11b70782f44b4bbe7135aced20a8d8, f84040238f8147d59e2d650ee4713639; no voters:
I20260925 22:40:52.136023 3402 raft_consensus.cc:2781] T bc4aee83e82f45c88da0f8c974ccb398 P f84040238f8147d59e2d650ee4713639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:40:52.136116 3402 raft_consensus.cc:482] T bc4aee83e82f45c88da0f8c974ccb398 P f84040238f8147d59e2d650ee4713639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:40:52.136162 3402 raft_consensus.cc:3037] T bc4aee83e82f45c88da0f8c974ccb398 P f84040238f8147d59e2d650ee4713639 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:52.137288 3402 raft_consensus.cc:504] T bc4aee83e82f45c88da0f8c974ccb398 P f84040238f8147d59e2d650ee4713639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de156a260db8484ba7ab55a3d32fcba5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37235 } } peers { permanent_uuid: "5d11b70782f44b4bbe7135aced20a8d8" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43327 } } peers { permanent_uuid: "f84040238f8147d59e2d650ee4713639" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 44171 } }
I20260925 22:40:52.137677 3402 leader_election.cc:288] T bc4aee83e82f45c88da0f8c974ccb398 P f84040238f8147d59e2d650ee4713639 [CANDIDATE]: Term 1 election: Requested vote from peers de156a260db8484ba7ab55a3d32fcba5 (127.2.74.194:37235), 5d11b70782f44b4bbe7135aced20a8d8 (127.2.74.193:43327)
I20260925 22:40:52.138043 3197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc4aee83e82f45c88da0f8c974ccb398" candidate_uuid: "f84040238f8147d59e2d650ee4713639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de156a260db8484ba7ab55a3d32fcba5"
I20260925 22:40:52.138085 3063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc4aee83e82f45c88da0f8c974ccb398" candidate_uuid: "f84040238f8147d59e2d650ee4713639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d11b70782f44b4bbe7135aced20a8d8"
I20260925 22:40:52.138213 3197 raft_consensus.cc:3037] T bc4aee83e82f45c88da0f8c974ccb398 P de156a260db8484ba7ab55a3d32fcba5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:52.138213 3063 raft_consensus.cc:3037] T bc4aee83e82f45c88da0f8c974ccb398 P 5d11b70782f44b4bbe7135aced20a8d8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:52.139756 3063 raft_consensus.cc:2454] T bc4aee83e82f45c88da0f8c974ccb398 P 5d11b70782f44b4bbe7135aced20a8d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f84040238f8147d59e2d650ee4713639 in term 1.
I20260925 22:40:52.139756 3197 raft_consensus.cc:2454] T bc4aee83e82f45c88da0f8c974ccb398 P de156a260db8484ba7ab55a3d32fcba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f84040238f8147d59e2d650ee4713639 in term 1.
I20260925 22:40:52.140163 3264 leader_election.cc:302] T bc4aee83e82f45c88da0f8c974ccb398 P f84040238f8147d59e2d650ee4713639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de156a260db8484ba7ab55a3d32fcba5, f84040238f8147d59e2d650ee4713639; no voters:
I20260925 22:40:52.140395 3402 raft_consensus.cc:2781] T bc4aee83e82f45c88da0f8c974ccb398 P f84040238f8147d59e2d650ee4713639 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:52.140654 3402 raft_consensus.cc:686] T bc4aee83e82f45c88da0f8c974ccb398 P f84040238f8147d59e2d650ee4713639 [term 1 LEADER]: Becoming Leader. State: Replica: f84040238f8147d59e2d650ee4713639, State: Running, Role: LEADER
I20260925 22:40:52.141122 3402 consensus_queue.cc:237] T bc4aee83e82f45c88da0f8c974ccb398 P f84040238f8147d59e2d650ee4713639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de156a260db8484ba7ab55a3d32fcba5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37235 } } peers { permanent_uuid: "5d11b70782f44b4bbe7135aced20a8d8" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43327 } } peers { permanent_uuid: "f84040238f8147d59e2d650ee4713639" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 44171 } }
I20260925 22:40:52.144949 2916 catalog_manager.cc:5495] T bc4aee83e82f45c88da0f8c974ccb398 P f84040238f8147d59e2d650ee4713639 reported cstate change: term changed from 0 to 1, leader changed from <none> to f84040238f8147d59e2d650ee4713639 (127.2.74.195). New cstate: current_term: 1 leader_uuid: "f84040238f8147d59e2d650ee4713639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de156a260db8484ba7ab55a3d32fcba5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d11b70782f44b4bbe7135aced20a8d8" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f84040238f8147d59e2d650ee4713639" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 44171 } health_report { overall_health: HEALTHY } } }
Sep 25 22:40:52 dist-test-slave-csnl krb5kdc[2872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.193@KRBTEST.COM
I20260925 22:40:52.166132 3391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:52.156247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45186 (local address 127.2.74.193:43327)
0925 22:40:52.156402 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:52.156408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:52.156593 (+ 185us) server_negotiation.cc:407] Connection header received
0925 22:40:52.156762 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:52.156770 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:52.156816 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:52.156907 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:52.157621 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.158251 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.159282 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.159527 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.159645 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:52.162351 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:52.162366 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:52.162383 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:52.162423 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:52.164185 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:52.164711 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:52.164717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:52.164719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:52.164809 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:52.165120 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:52.165124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:52.165125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:52.165406 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:52.165643 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:52.165817 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 22:40:52.165954 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 22:40:52 dist-test-slave-csnl krb5kdc[2872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376052, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:40:52 dist-test-slave-csnl krb5kdc[2872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376052, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.74.193@KRBTEST.COM
I20260925 22:40:52.194734 3391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:52.185882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45192 (local address 127.2.74.193:43327)
0925 22:40:52.186095 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:52.186101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:52.186242 (+ 141us) server_negotiation.cc:407] Connection header received
0925 22:40:52.186427 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:52.186430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:52.186474 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:52.186571 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:52.187359 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.187899 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.188763 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.188998 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.189157 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:52.191692 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:52.191695 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:52.191697 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:52.191723 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:52.193060 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:52.193477 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:52.193480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:52.193481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:52.193553 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:52.193795 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:52.193797 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:52.193798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:52.193974 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:52.194073 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:52.194382 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 22:40:52.194541 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
Sep 25 22:40:52 dist-test-slave-csnl krb5kdc[2872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:40:52 dist-test-slave-csnl krb5kdc[2872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.193@KRBTEST.COM
I20260925 22:40:52.220860 3391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:52.212539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45208 (local address 127.2.74.193:43327)
0925 22:40:52.212706 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:52.212711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:52.212761 (+ 50us) server_negotiation.cc:407] Connection header received
0925 22:40:52.212931 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:52.212933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:52.212986 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:52.213056 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:52.213749 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.214259 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.215142 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:52.215355 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:52.215443 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:52.217864 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:52.217868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:52.217870 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:52.217895 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:52.219397 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:52.219804 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:52.219806 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:52.219807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:52.219876 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:52.220124 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:52.220126 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:52.220127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:52.220282 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:52.220378 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:52.220574 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 22:40:52.220669 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 22:40:52.222337 3008 tablet_service.cc:2908] Rejecting scan request for tablet bc4aee83e82f45c88da0f8c974ccb398: Uninitialized: clean time has not yet been initialized
I20260925 22:40:52.224864 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 2980
I20260925 22:40:52.230899 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 3114
I20260925 22:40:52.236565 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 3248
I20260925 22:40:52.242925 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 2885
2026-09-25T22:40:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2353 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:40:52 dist-test-slave-csnl krb5kdc[3418](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:40:52 dist-test-slave-csnl krb5kdc[3418](info): set up 2 sockets
Sep 25 22:40:52 dist-test-slave-csnl krb5kdc[3418](info): commencing operation
krb5kdc: starting...
W20260925 22:40:53.286725 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:40:53 dist-test-slave-csnl krb5kdc[3418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:40:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:40:53.418195 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:38813
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:43193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:38813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:53.534682 3431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:53.535079 3431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:53.535188 3431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:53.539426 3431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:53.539546 3431 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 22:40:53.539579 3431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:53.539621 3431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:53.539659 3431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:53.539695 3431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:53.545320 3431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:43193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:38813
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:38813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:53.546525 3431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:53.547657 3431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:53.556815 3437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:53.556815 3436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:53.556808 3439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:53.556885 3431 server_base.cc:1034] running on GCE node
I20260925 22:40:53.557827 3431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:53.559124 3431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:53.560307 3431 hybrid_clock.cc:648] HybridClock initialized: now 1790376053560293 us; error 36 us; skew 500 ppm
Sep 25 22:40:53 dist-test-slave-csnl krb5kdc[3418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376053, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:53.563359 3431 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:40:53.564455 3431 webserver.cc:460] Webserver started at http://127.2.74.254:45395/ using document root <none> and password file <none>
I20260925 22:40:53.565091 3431 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:53.565162 3431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:53.565403 3431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:53.567759 3431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "49e3aa4e21444fbcac571579b544ff97"
format_stamp: "Formatted at 2026-09-25 22:40:53 on dist-test-slave-csnl"
server_key: "527080c25e041ba822489b1b74155084"
server_key_iv: "f8f385edc676dc5a82f7d94e180a51fc"
server_key_version: "encryptionkey@0"
I20260925 22:40:53.568390 3431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "49e3aa4e21444fbcac571579b544ff97"
format_stamp: "Formatted at 2026-09-25 22:40:53 on dist-test-slave-csnl"
server_key: "527080c25e041ba822489b1b74155084"
server_key_iv: "f8f385edc676dc5a82f7d94e180a51fc"
server_key_version: "encryptionkey@0"
I20260925 22:40:53.572701 3431 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:40:53.575657 3446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:53.577248 3431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:40:53.577446 3431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "49e3aa4e21444fbcac571579b544ff97"
format_stamp: "Formatted at 2026-09-25 22:40:53 on dist-test-slave-csnl"
server_key: "527080c25e041ba822489b1b74155084"
server_key_iv: "f8f385edc676dc5a82f7d94e180a51fc"
server_key_version: "encryptionkey@0"
I20260925 22:40:53.577593 3431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:53.601858 3431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:53.605355 3431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:53.605593 3431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:53.614655 3431 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:38813
I20260925 22:40:53.614689 3508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:38813 every 8 connection(s)
I20260925 22:40:53.615824 3431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:53.618723 3509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:53.625128 3509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49e3aa4e21444fbcac571579b544ff97: Bootstrap starting.
I20260925 22:40:53.625679 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 3431
I20260925 22:40:53.625797 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:40:53.626045 2347 external_mini_cluster.cc:1428] Setting key 785aaae8742e31820862b1315e3f7aae
I20260925 22:40:53.628346 3509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49e3aa4e21444fbcac571579b544ff97: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:53.629185 3509 log.cc:824] T 00000000000000000000000000000000 P 49e3aa4e21444fbcac571579b544ff97: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:53.631953 3509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49e3aa4e21444fbcac571579b544ff97: No bootstrap required, opened a new log
Sep 25 22:40:53 dist-test-slave-csnl krb5kdc[3418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:40:53.636981 3509 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49e3aa4e21444fbcac571579b544ff97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e3aa4e21444fbcac571579b544ff97" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38813 } }
I20260925 22:40:53.637387 3509 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49e3aa4e21444fbcac571579b544ff97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:53.637470 3509 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49e3aa4e21444fbcac571579b544ff97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e3aa4e21444fbcac571579b544ff97, State: Initialized, Role: FOLLOWER
I20260925 22:40:53.638049 3509 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49e3aa4e21444fbcac571579b544ff97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e3aa4e21444fbcac571579b544ff97" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38813 } }
I20260925 22:40:53.638240 3509 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49e3aa4e21444fbcac571579b544ff97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:40:53.638315 3509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49e3aa4e21444fbcac571579b544ff97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:40:53.638447 3509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49e3aa4e21444fbcac571579b544ff97 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:53.639526 3509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49e3aa4e21444fbcac571579b544ff97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e3aa4e21444fbcac571579b544ff97" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38813 } }
I20260925 22:40:53.639920 3509 leader_election.cc:302] T 00000000000000000000000000000000 P 49e3aa4e21444fbcac571579b544ff97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49e3aa4e21444fbcac571579b544ff97; no voters:
I20260925 22:40:53.640372 3509 leader_election.cc:288] T 00000000000000000000000000000000 P 49e3aa4e21444fbcac571579b544ff97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:40:53.640487 3514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49e3aa4e21444fbcac571579b544ff97 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:53.640746 3514 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49e3aa4e21444fbcac571579b544ff97 [term 1 LEADER]: Becoming Leader. State: Replica: 49e3aa4e21444fbcac571579b544ff97, State: Running, Role: LEADER
I20260925 22:40:53.641178 3514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49e3aa4e21444fbcac571579b544ff97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e3aa4e21444fbcac571579b544ff97" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38813 } }
I20260925 22:40:53.641960 3509 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49e3aa4e21444fbcac571579b544ff97 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:53.643224 3516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49e3aa4e21444fbcac571579b544ff97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49e3aa4e21444fbcac571579b544ff97. Latest consensus state: current_term: 1 leader_uuid: "49e3aa4e21444fbcac571579b544ff97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e3aa4e21444fbcac571579b544ff97" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38813 } } }
I20260925 22:40:53.643684 3516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49e3aa4e21444fbcac571579b544ff97 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:53.643572 3515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49e3aa4e21444fbcac571579b544ff97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49e3aa4e21444fbcac571579b544ff97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e3aa4e21444fbcac571579b544ff97" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38813 } } }
I20260925 22:40:53.643961 3512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:53.627736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52222 (local address 127.2.74.254:38813)
0925 22:40:53.628260 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:53.628275 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:53.628313 (+ 38us) server_negotiation.cc:407] Connection header received
0925 22:40:53.629091 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:53.629112 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:53.629549 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:53.629992 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:53.631500 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:53.632624 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:53.633950 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:53.634470 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:53.634617 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:53.637867 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:53.637873 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:53.637890 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:53.637926 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:53.640001 (+ 2075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:53.640641 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:53.640649 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:53.640657 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:53.640824 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:53.641251 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:53.641255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:53.641256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:53.641821 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:53.642046 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:53.642389 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 22:40:53.642738 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":168,"threads_started":1}
I20260925 22:40:53.644045 3515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49e3aa4e21444fbcac571579b544ff97 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:53.644295 3527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:53.648406 3527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:40:53.654680 3527 catalog_manager.cc:1347] Generated new cluster ID: cebe9e0fa690414cad8552c639452616
I20260925 22:40:53.654773 3527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:53.706864 3527 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:53.708086 3527 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:53.725231 3527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49e3aa4e21444fbcac571579b544ff97: Generated new TSK 0
I20260925 22:40:53.726047 3527 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
I20260925 22:40:53.764323 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:0
--local_ip_for_outbound_sockets=127.2.74.193
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:38813
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:43193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:53.881861 3535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:53.882157 3535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:53.882246 3535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:53.886596 3535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:53.886698 3535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:53.886740 3535 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:40:53.886816 3535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:40:53.892665 3535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:43193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.74.254:38813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:53.893884 3535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:53.895036 3535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:53.905325 3540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:53.905359 3543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:53.905411 3541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:53.905642 3535 server_base.cc:1034] running on GCE node
I20260925 22:40:53.906207 3535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:53.907008 3535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:53.908215 3535 hybrid_clock.cc:648] HybridClock initialized: now 1790376053908200 us; error 38 us; skew 500 ppm
Sep 25 22:40:53 dist-test-slave-csnl krb5kdc[3418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376053, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:53.911286 3535 init.cc:377] Logged in from keytab as kudu/127.2.74.193@KRBTEST.COM (short username kudu)
I20260925 22:40:53.912236 3535 webserver.cc:460] Webserver started at http://127.2.74.193:45673/ using document root <none> and password file <none>
I20260925 22:40:53.912801 3535 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:53.912865 3535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:53.913036 3535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:53.915318 3535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-0/data/instance:
uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0"
format_stamp: "Formatted at 2026-09-25 22:40:53 on dist-test-slave-csnl"
server_key: "64a70f7e30b3922cef1797272a798899"
server_key_iv: "e7abe3b0f75a80f0d75b11b91ef40ac6"
server_key_version: "encryptionkey@0"
I20260925 22:40:53.915877 3535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance:
uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0"
format_stamp: "Formatted at 2026-09-25 22:40:53 on dist-test-slave-csnl"
server_key: "64a70f7e30b3922cef1797272a798899"
server_key_iv: "e7abe3b0f75a80f0d75b11b91ef40ac6"
server_key_version: "encryptionkey@0"
I20260925 22:40:53.919857 3535 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260925 22:40:53.922526 3550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:53.923930 3535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:40:53.924048 3535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0"
format_stamp: "Formatted at 2026-09-25 22:40:53 on dist-test-slave-csnl"
server_key: "64a70f7e30b3922cef1797272a798899"
server_key_iv: "e7abe3b0f75a80f0d75b11b91ef40ac6"
server_key_version: "encryptionkey@0"
I20260925 22:40:53.924165 3535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:53.937827 3535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:53.941406 3535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:53.941576 3535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:53.943414 3535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:53.943502 3535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:53.943572 3535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:53.943603 3535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:40:53 dist-test-slave-csnl krb5kdc[3418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376053, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:40:53.954648 3535 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:40695
I20260925 22:40:53.954766 3665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:40695 every 8 connection(s)
I20260925 22:40:53.955679 3535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:40:53.957439 3512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:53.944202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:54049 (local address 127.2.74.254:38813)
0925 22:40:53.944366 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:53.944374 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:53.945333 (+ 959us) server_negotiation.cc:407] Connection header received
0925 22:40:53.946378 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:53.946382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:53.946451 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:53.946498 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:53.948188 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:53.948779 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:53.949977 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:53.950229 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:53.950350 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:53.953639 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:53.953644 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:53.953647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:53.953682 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:53.955313 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:53.955881 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:53.955884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:53.955885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:53.955957 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:53.956342 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:53.956345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:53.956347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:53.956562 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:53.956673 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:53.957147 (+ 474us) server_negotiation.cc:299] Negotiation successful
0925 22:40:53.957272 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:40:53.958191 3559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:53.944682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:38813 (local address 127.2.74.193:54049)
0925 22:40:53.945154 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:53.945206 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:53.946106 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:53.946838 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:53.946845 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:53.947324 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:53.948015 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:53.948038 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:53.948914 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:53.948918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:53.949843 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:53.949853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:53.950405 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:53.950409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:53.950598 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:53.951314 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:53.951339 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:53.953455 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:53.955446 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:53.955451 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:53.955467 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:53.955733 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:53.956056 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:53.956059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:53.956061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:53.956232 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:53.956669 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:53.956675 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:53.956913 (+ 238us) client_negotiation.cc:769] Sending connection context
0925 22:40:53.957096 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 22:40:53.957406 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":131,"threads_started":1}
I20260925 22:40:53.959408 3666 heartbeater.cc:347] Connected to a master server at 127.2.74.254:38813
I20260925 22:40:53.959548 3666 heartbeater.cc:464] Registering TS with master...
I20260925 22:40:53.960119 3666 heartbeater.cc:511] Master 127.2.74.254:38813 requested a full tablet report, sending...
I20260925 22:40:53.961408 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 3535
I20260925 22:40:53.961514 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance
I20260925 22:40:53.961804 2347 external_mini_cluster.cc:1428] Setting key 4e8d25541a99b806c53dbd0d0053a2b3
I20260925 22:40:53.961920 3462 ts_manager.cc:194] Registered new tserver with Master: d18a7a96e02f4f4fb960ffb0c8cbfbb0 (127.2.74.193:40695)
I20260925 22:40:53.963498 3462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.193@KRBTEST.COM'} at 127.2.74.193:54049
WARNING: no policy specified for kudu/127.2.74.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.194@KRBTEST.COM" created.
I20260925 22:40:53.974864 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:53.961797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:38813 (local address 127.2.74.193:35231)
0925 22:40:53.962380 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:53.962393 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:53.962507 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:53.962784 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:53.962787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:53.963049 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:53.963184 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:53.963192 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:53.963992 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:53.963995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:53.964854 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:53.964868 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:53.965454 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:53.965459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:53.965579 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:53.971007 (+ 5428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:53.971033 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:53.971389 (+ 356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:53.973262 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:53.973265 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:53.973267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:53.973466 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:53.973777 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:53.973782 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:53.973784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:53.973872 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:53.974278 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:53.974281 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:53.974390 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 22:40:53.974498 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 22:40:53.974676 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":101,"threads_started":1}
I20260925 22:40:53.974859 3512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:53.961846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:35231 (local address 127.2.74.254:38813)
0925 22:40:53.962014 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:53.962018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:53.962463 (+ 445us) server_negotiation.cc:407] Connection header received
0925 22:40:53.962606 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:53.962609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:53.962662 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:53.962750 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:53.963288 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:53.963864 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:53.965020 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:53.965327 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:53.965438 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:53.971599 (+ 6161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:53.971603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:53.971605 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:53.971630 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:53.972928 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:53.973593 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:53.973598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:53.973600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:53.973682 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:53.973962 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:53.973966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:53.973968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:53.974178 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:53.974290 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:53.974564 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 22:40:53.974680 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
I20260925 22:40:53.990159 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:0
--local_ip_for_outbound_sockets=127.2.74.194
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:38813
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:43193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:54.107483 3675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:54.107810 3675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:54.107874 3675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:54.112105 3675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:54.112177 3675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:54.112218 3675 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:40:54.112253 3675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:40:54.117694 3675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:43193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.74.254:38813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:54.118773 3675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:54.119896 3675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:54.130560 3681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:54.130558 3680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:54.130509 3683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:54.131006 3675 server_base.cc:1034] running on GCE node
I20260925 22:40:54.131435 3675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:54.132112 3675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:54.133282 3675 hybrid_clock.cc:648] HybridClock initialized: now 1790376054133256 us; error 42 us; skew 500 ppm
Sep 25 22:40:54 dist-test-slave-csnl krb5kdc[3418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376054, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:54.136368 3675 init.cc:377] Logged in from keytab as kudu/127.2.74.194@KRBTEST.COM (short username kudu)
I20260925 22:40:54.137372 3675 webserver.cc:460] Webserver started at http://127.2.74.194:44621/ using document root <none> and password file <none>
I20260925 22:40:54.138008 3675 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:54.138116 3675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:54.138368 3675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:54.140728 3675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-1/data/instance:
uuid: "022f7ddabcfc42c790f5c08e603c922b"
format_stamp: "Formatted at 2026-09-25 22:40:54 on dist-test-slave-csnl"
server_key: "a8729b4097bc149ef31d5d701a3602ed"
server_key_iv: "828cf4636a0a75c1f9809711bd099aca"
server_key_version: "encryptionkey@0"
I20260925 22:40:54.141418 3675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance:
uuid: "022f7ddabcfc42c790f5c08e603c922b"
format_stamp: "Formatted at 2026-09-25 22:40:54 on dist-test-slave-csnl"
server_key: "a8729b4097bc149ef31d5d701a3602ed"
server_key_iv: "828cf4636a0a75c1f9809711bd099aca"
server_key_version: "encryptionkey@0"
I20260925 22:40:54.145991 3675 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 22:40:54.149168 3690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:54.150616 3675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:40:54.150818 3675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "022f7ddabcfc42c790f5c08e603c922b"
format_stamp: "Formatted at 2026-09-25 22:40:54 on dist-test-slave-csnl"
server_key: "a8729b4097bc149ef31d5d701a3602ed"
server_key_iv: "828cf4636a0a75c1f9809711bd099aca"
server_key_version: "encryptionkey@0"
I20260925 22:40:54.150995 3675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:54.166378 3675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:54.171125 3675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:54.171370 3675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:54.172922 3675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:54.173015 3675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:54.173101 3675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:54.173149 3675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:40:54 dist-test-slave-csnl krb5kdc[3418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376054, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:40:54.185392 3675 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:35683
I20260925 22:40:54.185514 3805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:35683 every 8 connection(s)
I20260925 22:40:54.186726 3675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:40:54.187146 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 3675
I20260925 22:40:54.187264 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance
I20260925 22:40:54.187561 2347 external_mini_cluster.cc:1428] Setting key 8258b16abd963eb4d937775a301c28c7
I20260925 22:40:54.189317 3512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:54.174034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:41023 (local address 127.2.74.254:38813)
0925 22:40:54.174185 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:54.174189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:54.175388 (+ 1199us) server_negotiation.cc:407] Connection header received
0925 22:40:54.176695 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:54.176699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:54.176777 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:54.176886 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:54.178698 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.179409 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.180840 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.181242 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.181362 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:54.184563 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:54.184575 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:54.184578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:54.184615 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:54.186494 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:54.187325 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:54.187329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:54.187332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:54.187442 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:54.187908 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:54.187914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:54.187919 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:54.188158 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:54.188288 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:54.188972 (+ 684us) server_negotiation.cc:299] Negotiation successful
0925 22:40:54.189154 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 22:40:54.190164 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:54.174687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:38813 (local address 127.2.74.194:41023)
0925 22:40:54.175223 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:54.175270 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:54.176377 (+ 1107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:54.177131 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:54.177139 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:54.177607 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:54.178510 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:54.178534 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.179566 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.179570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:54.180654 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:54.180670 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.181426 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.181430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:54.181645 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:54.182422 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:54.182441 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:54.184317 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:54.186652 (+ 2335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:54.186658 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:54.186672 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:54.187150 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:54.187580 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:54.187585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:54.187587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:54.187781 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:54.188285 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:54.188292 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:54.188628 (+ 336us) client_negotiation.cc:769] Sending connection context
0925 22:40:54.188851 (+ 223us) client_negotiation.cc:240] Negotiation successful
0925 22:40:54.189181 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":156,"threads_started":1}
I20260925 22:40:54.192015 3806 heartbeater.cc:347] Connected to a master server at 127.2.74.254:38813
I20260925 22:40:54.192220 3806 heartbeater.cc:464] Registering TS with master...
I20260925 22:40:54.192759 3806 heartbeater.cc:511] Master 127.2.74.254:38813 requested a full tablet report, sending...
I20260925 22:40:54.193988 3463 ts_manager.cc:194] Registered new tserver with Master: 022f7ddabcfc42c790f5c08e603c922b (127.2.74.194:35683)
I20260925 22:40:54.194840 3463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.194@KRBTEST.COM'} at 127.2.74.194:41023
WARNING: no policy specified for kudu/127.2.74.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.195@KRBTEST.COM" created.
I20260925 22:40:54.202678 3512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:54.195203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:47655 (local address 127.2.74.254:38813)
0925 22:40:54.195334 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:54.195339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:54.195352 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:40:54.195390 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:54.195393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:54.195436 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:54.195522 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:54.196169 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.196671 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.197516 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.197748 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.197858 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:54.199110 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:54.199115 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:54.199123 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:54.199145 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:54.200634 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:54.201175 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:54.201179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:54.201181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:54.201308 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:54.201668 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:54.201671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:54.201672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:54.201972 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:54.202109 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:54.202436 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 22:40:54.202553 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:40:54.202772 3813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:54.194643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:38813 (local address 127.2.74.194:47655)
0925 22:40:54.195041 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:54.195058 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:54.195164 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:54.195576 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:54.195583 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:54.195848 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:54.196034 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:54.196047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.196803 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.196806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:54.197384 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:54.197392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.197866 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.197869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:54.197965 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:54.198546 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:54.198559 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:54.198909 (+ 350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:54.200784 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:54.200788 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:54.200790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:54.201034 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:54.201421 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:54.201425 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:54.201427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:54.201528 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:54.202113 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:54.202119 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:54.202237 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 22:40:54.202330 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 22:40:54.202519 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
I20260925 22:40:54.218593 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:0
--local_ip_for_outbound_sockets=127.2.74.195
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:38813
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:43193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:54.337859 3815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:54.338186 3815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:54.338232 3815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:54.342599 3815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:54.342682 3815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:54.342785 3815 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:40:54.342833 3815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:40:54.349325 3815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:43193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.74.254:38813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:54.350490 3815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:54.351639 3815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:54.362537 3821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:54.362524 3823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:54.362524 3820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:54.362900 3815 server_base.cc:1034] running on GCE node
I20260925 22:40:54.363404 3815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:54.364104 3815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:54.365347 3815 hybrid_clock.cc:648] HybridClock initialized: now 1790376054365327 us; error 39 us; skew 500 ppm
Sep 25 22:40:54 dist-test-slave-csnl krb5kdc[3418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376054, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:54.368546 3815 init.cc:377] Logged in from keytab as kudu/127.2.74.195@KRBTEST.COM (short username kudu)
I20260925 22:40:54.369560 3815 webserver.cc:460] Webserver started at http://127.2.74.195:33197/ using document root <none> and password file <none>
I20260925 22:40:54.370137 3815 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:54.370204 3815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:54.370380 3815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:54.372676 3815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-2/data/instance:
uuid: "56baba7fcf0c48f3994957d47649dc55"
format_stamp: "Formatted at 2026-09-25 22:40:54 on dist-test-slave-csnl"
server_key: "c0a2f73e794034de7a99adc97c897b0e"
server_key_iv: "3d3d94fb794f97aaf99c855ae14537ee"
server_key_version: "encryptionkey@0"
I20260925 22:40:54.373245 3815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance:
uuid: "56baba7fcf0c48f3994957d47649dc55"
format_stamp: "Formatted at 2026-09-25 22:40:54 on dist-test-slave-csnl"
server_key: "c0a2f73e794034de7a99adc97c897b0e"
server_key_iv: "3d3d94fb794f97aaf99c855ae14537ee"
server_key_version: "encryptionkey@0"
I20260925 22:40:54.377357 3815 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:40:54.380141 3830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:54.381543 3815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:40:54.381762 3815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "56baba7fcf0c48f3994957d47649dc55"
format_stamp: "Formatted at 2026-09-25 22:40:54 on dist-test-slave-csnl"
server_key: "c0a2f73e794034de7a99adc97c897b0e"
server_key_iv: "3d3d94fb794f97aaf99c855ae14537ee"
server_key_version: "encryptionkey@0"
I20260925 22:40:54.381909 3815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:54.429133 3815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:54.432364 3815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:54.432612 3815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:54.434185 3815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:54.434278 3815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:54.434353 3815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:54.434397 3815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:40:54 dist-test-slave-csnl krb5kdc[3418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376054, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:40:54.445977 3815 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:33851
I20260925 22:40:54.446915 3815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:40:54.447160 3945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:33851 every 8 connection(s)
I20260925 22:40:54.450464 3512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:54.435272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:38645 (local address 127.2.74.254:38813)
0925 22:40:54.435405 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:54.435408 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:54.436682 (+ 1274us) server_negotiation.cc:407] Connection header received
0925 22:40:54.438060 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:54.438090 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:54.438161 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:54.438242 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:54.439909 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.440820 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.442676 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.443113 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.443231 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:54.446613 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:54.446617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:54.446619 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:54.446647 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:54.448374 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:54.448915 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:54.448919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:54.448921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:54.449019 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:54.449369 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:54.449374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:54.449376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:54.449571 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:54.449678 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:54.450174 (+ 496us) server_negotiation.cc:299] Negotiation successful
0925 22:40:54.450302 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:40:54.451416 3839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:54.435942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:38813 (local address 127.2.74.195:38645)
0925 22:40:54.436494 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:54.436542 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:54.437701 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:54.438698 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:54.438708 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:54.439222 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:54.439752 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:54.439767 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.440989 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.440993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:54.442392 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:54.442421 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.443307 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.443313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:54.443594 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:54.444417 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:54.444448 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:54.446383 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:54.448501 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:54.448507 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:54.448523 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:54.448798 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:54.449119 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:54.449124 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:54.449126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:54.449270 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:54.449676 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:54.449684 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:54.449936 (+ 252us) client_negotiation.cc:769] Sending connection context
0925 22:40:54.450133 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 22:40:54.450465 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":138,"threads_started":1}
I20260925 22:40:54.452643 3946 heartbeater.cc:347] Connected to a master server at 127.2.74.254:38813
I20260925 22:40:54.452807 3946 heartbeater.cc:464] Registering TS with master...
I20260925 22:40:54.453328 3946 heartbeater.cc:511] Master 127.2.74.254:38813 requested a full tablet report, sending...
I20260925 22:40:54.454419 3462 ts_manager.cc:194] Registered new tserver with Master: 56baba7fcf0c48f3994957d47649dc55 (127.2.74.195:33851)
I20260925 22:40:54.455332 3462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.195@KRBTEST.COM'} at 127.2.74.195:38645
I20260925 22:40:54.456353 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 3815
I20260925 22:40:54.456477 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance
I20260925 22:40:54.456833 2347 external_mini_cluster.cc:1428] Setting key ea88dd14536a1ef450b387e356a35124
I20260925 22:40:54.458990 2347 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:40:54.463693 3512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:54.455193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:46501 (local address 127.2.74.254:38813)
0925 22:40:54.455378 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:54.455382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:54.455535 (+ 153us) server_negotiation.cc:407] Connection header received
0925 22:40:54.455981 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:54.455985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:54.456033 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:54.456141 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:54.456823 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.457507 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.458354 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.458661 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.458768 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:54.460223 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:54.460230 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:54.460235 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:54.460269 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:54.461893 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:54.462366 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:54.462369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:54.462371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:54.462460 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:54.462741 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:54.462745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:54.462747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:54.463017 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:54.463132 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:54.463422 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 22:40:54.463533 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:40:54.463702 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:54.455138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:38813 (local address 127.2.74.195:46501)
0925 22:40:54.455458 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:54.455474 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:54.455581 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:54.456254 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:54.456258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:54.456479 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:54.456600 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:54.456608 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.457648 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.457651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:54.458225 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:54.458233 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.458894 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.458897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:54.459031 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:54.459676 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:54.459689 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:54.460078 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:54.462009 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:54.462012 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:54.462014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:54.462252 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:54.462554 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:54.462558 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:54.462560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:54.462648 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:54.463131 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:54.463134 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:54.463243 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 22:40:54.463354 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 22:40:54.463523 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":121,"threads_started":1}
Sep 25 22:40:54 dist-test-slave-csnl krb5kdc[3418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376054, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:40:54 dist-test-slave-csnl krb5kdc[3418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376054, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:40:54.486529 3512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:54.477600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52234 (local address 127.2.74.254:38813)
0925 22:40:54.477737 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:54.477740 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:54.477753 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:40:54.477793 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:54.477795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:54.477834 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:54.477902 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:54.478678 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.479364 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.480237 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.480457 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.480542 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:54.483257 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:54.483260 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:54.483263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:54.483300 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:54.484628 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:54.485093 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:54.485096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:54.485097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:54.485174 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:54.485402 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:54.485404 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:54.485405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:54.485584 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:54.485739 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:54.486145 (+ 406us) server_negotiation.cc:299] Negotiation successful
0925 22:40:54.486297 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:40:54.489703 3462 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:40:54.493219 3462 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:40:54.509261 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:54.504193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38940 (local address 127.2.74.194:35683)
0925 22:40:54.504629 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:54.504641 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:54.504664 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:40:54.504743 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:54.504749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:54.504951 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:54.505111 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:54.505348 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.506252 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.507558 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.508446 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.508576 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:54.508754 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:54.508981 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 22:40:54.509094 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":110,"threads_started":1}
I20260925 22:40:54.509259 3965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:54.503716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:35683 (local address 127.0.0.1:38940)
0925 22:40:54.504308 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:54.504324 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:54.504422 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:54.505110 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:54.505113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:54.505132 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:54.505226 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:54.505231 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.506373 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.506375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:54.507428 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:54.507438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.508603 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.508607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:54.508747 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:54.508795 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 22:40:54.508922 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 22:40:54.509061 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":23,"thread_start_us":125,"threads_started":1}
I20260925 22:40:54.509272 3963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:54.502508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:40695 (local address 127.0.0.1:34358)
0925 22:40:54.502896 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:54.502976 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:54.503113 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:54.504527 (+ 1414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:54.504533 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:54.504556 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:54.504682 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:54.504687 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.506037 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.506041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:54.507570 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:54.507581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.508807 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.508811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:54.508932 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:54.508956 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:40:54.509029 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 22:40:54.509099 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":44,"thread_start_us":140,"threads_started":1}
I20260925 22:40:54.509618 3968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:54.504208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50474 (local address 127.2.74.195:33851)
0925 22:40:54.504669 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:54.504681 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:54.504700 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:40:54.504759 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:54.504763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:54.504950 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:54.505111 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:54.505540 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.506620 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.508017 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.508635 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.508749 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:54.508913 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:54.509340 (+ 427us) server_negotiation.cc:299] Negotiation successful
0925 22:40:54.509457 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":107,"threads_started":1}
I20260925 22:40:54.509913 3966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:54.504049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:33851 (local address 127.0.0.1:50474)
0925 22:40:54.504359 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:54.504381 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:54.504475 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:54.505264 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:54.505267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:54.505315 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:54.505405 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:54.505411 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.506754 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.506757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:54.507893 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:54.507900 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.508973 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.508976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:54.509057 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:54.509078 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:40:54.509146 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 22:40:54.509213 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":23,"thread_start_us":98,"threads_started":1}
I20260925 22:40:54.510172 3964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:54.502636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34358 (local address 127.2.74.193:40695)
0925 22:40:54.503436 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:54.503444 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:54.503479 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:40:54.503563 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:54.503571 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:54.503810 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:54.504027 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:54.504804 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.505903 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.507712 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.508460 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.508576 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:54.508740 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:54.509923 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0925 22:40:54.510046 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":96,"threads_started":1}
I20260925 22:40:54.512774 3740 tablet_service.cc:1467] Processing CreateTablet for tablet 73c792a2cdc14d438b3b08abea585bd7 (DEFAULT_TABLE table=test-table [id=6cd3afaff39c4d2897fd2c510d33fb54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:54.512784 3880 tablet_service.cc:1467] Processing CreateTablet for tablet 73c792a2cdc14d438b3b08abea585bd7 (DEFAULT_TABLE table=test-table [id=6cd3afaff39c4d2897fd2c510d33fb54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:54.513224 3600 tablet_service.cc:1467] Processing CreateTablet for tablet 73c792a2cdc14d438b3b08abea585bd7 (DEFAULT_TABLE table=test-table [id=6cd3afaff39c4d2897fd2c510d33fb54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:54.514026 3880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73c792a2cdc14d438b3b08abea585bd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:54.514029 3740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73c792a2cdc14d438b3b08abea585bd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:54.514155 3600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73c792a2cdc14d438b3b08abea585bd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:54.523039 3969 tablet_bootstrap.cc:492] T 73c792a2cdc14d438b3b08abea585bd7 P 022f7ddabcfc42c790f5c08e603c922b: Bootstrap starting.
I20260925 22:40:54.523983 3970 tablet_bootstrap.cc:492] T 73c792a2cdc14d438b3b08abea585bd7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0: Bootstrap starting.
I20260925 22:40:54.525818 3969 tablet_bootstrap.cc:654] T 73c792a2cdc14d438b3b08abea585bd7 P 022f7ddabcfc42c790f5c08e603c922b: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:54.526326 3970 tablet_bootstrap.cc:654] T 73c792a2cdc14d438b3b08abea585bd7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:54.526794 3969 log.cc:824] T 73c792a2cdc14d438b3b08abea585bd7 P 022f7ddabcfc42c790f5c08e603c922b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:54.527247 3970 log.cc:824] T 73c792a2cdc14d438b3b08abea585bd7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:54.527696 3971 tablet_bootstrap.cc:492] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55: Bootstrap starting.
I20260925 22:40:54.529249 3970 tablet_bootstrap.cc:492] T 73c792a2cdc14d438b3b08abea585bd7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0: No bootstrap required, opened a new log
I20260925 22:40:54.529259 3969 tablet_bootstrap.cc:492] T 73c792a2cdc14d438b3b08abea585bd7 P 022f7ddabcfc42c790f5c08e603c922b: No bootstrap required, opened a new log
I20260925 22:40:54.529469 3970 ts_tablet_manager.cc:1397] T 73c792a2cdc14d438b3b08abea585bd7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:40:54.529471 3969 ts_tablet_manager.cc:1397] T 73c792a2cdc14d438b3b08abea585bd7 P 022f7ddabcfc42c790f5c08e603c922b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 22:40:54.530104 3971 tablet_bootstrap.cc:654] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:54.531071 3971 log.cc:824] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:54.533581 3971 tablet_bootstrap.cc:492] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55: No bootstrap required, opened a new log
I20260925 22:40:54.533517 3970 raft_consensus.cc:348] T 73c792a2cdc14d438b3b08abea585bd7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } }
I20260925 22:40:54.533517 3969 raft_consensus.cc:348] T 73c792a2cdc14d438b3b08abea585bd7 P 022f7ddabcfc42c790f5c08e603c922b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } }
I20260925 22:40:54.533805 3971 ts_tablet_manager.cc:1397] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:40:54.533838 3969 raft_consensus.cc:374] T 73c792a2cdc14d438b3b08abea585bd7 P 022f7ddabcfc42c790f5c08e603c922b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:54.533836 3970 raft_consensus.cc:374] T 73c792a2cdc14d438b3b08abea585bd7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:54.533888 3969 raft_consensus.cc:729] T 73c792a2cdc14d438b3b08abea585bd7 P 022f7ddabcfc42c790f5c08e603c922b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 022f7ddabcfc42c790f5c08e603c922b, State: Initialized, Role: FOLLOWER
I20260925 22:40:54.533891 3970 raft_consensus.cc:729] T 73c792a2cdc14d438b3b08abea585bd7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d18a7a96e02f4f4fb960ffb0c8cbfbb0, State: Initialized, Role: FOLLOWER
I20260925 22:40:54.534374 3970 consensus_queue.cc:260] T 73c792a2cdc14d438b3b08abea585bd7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } }
I20260925 22:40:54.534406 3969 consensus_queue.cc:260] T 73c792a2cdc14d438b3b08abea585bd7 P 022f7ddabcfc42c790f5c08e603c922b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } }
I20260925 22:40:54.535110 3666 heartbeater.cc:503] Master 127.2.74.254:38813 was elected leader, sending a full tablet report...
I20260925 22:40:54.535487 3970 ts_tablet_manager.cc:1428] T 73c792a2cdc14d438b3b08abea585bd7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:40:54.535494 3969 ts_tablet_manager.cc:1428] T 73c792a2cdc14d438b3b08abea585bd7 P 022f7ddabcfc42c790f5c08e603c922b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:40:54.535840 3806 heartbeater.cc:503] Master 127.2.74.254:38813 was elected leader, sending a full tablet report...
I20260925 22:40:54.537744 3971 raft_consensus.cc:348] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } }
I20260925 22:40:54.538089 3971 raft_consensus.cc:374] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:54.538163 3971 raft_consensus.cc:729] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56baba7fcf0c48f3994957d47649dc55, State: Initialized, Role: FOLLOWER
I20260925 22:40:54.538684 3971 consensus_queue.cc:260] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } }
I20260925 22:40:54.539515 3946 heartbeater.cc:503] Master 127.2.74.254:38813 was elected leader, sending a full tablet report...
I20260925 22:40:54.539911 3971 ts_tablet_manager.cc:1428] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
W20260925 22:40:54.688339 3807 tablet.cc:2369] T 73c792a2cdc14d438b3b08abea585bd7 P 022f7ddabcfc42c790f5c08e603c922b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:40:54.698036 3947 tablet.cc:2369] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:40:54.707106 3667 tablet.cc:2369] T 73c792a2cdc14d438b3b08abea585bd7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:40:54.798517 3977 raft_consensus.cc:482] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:40:54.798759 3977 raft_consensus.cc:504] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } }
I20260925 22:40:54.799983 3977 leader_election.cc:288] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d18a7a96e02f4f4fb960ffb0c8cbfbb0 (127.2.74.193:40695), 022f7ddabcfc42c790f5c08e603c922b (127.2.74.194:35683)
I20260925 22:40:54.804745 3964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:54.800416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:60137 (local address 127.2.74.193:40695)
0925 22:40:54.800608 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:54.800613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:54.800739 (+ 126us) server_negotiation.cc:407] Connection header received
0925 22:40:54.800822 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:54.800825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:54.800878 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:54.800969 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:54.801308 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.801927 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.803447 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.804118 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.804256 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:54.804317 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:54.804525 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 22:40:54.804593 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260925 22:40:54.804764 3978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:54.800285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:40695 (local address 127.2.74.195:60137)
0925 22:40:54.800637 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:54.800651 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:54.800744 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:54.800981 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:54.800984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:54.801025 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:54.801188 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:54.801195 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.802038 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.802041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:54.803306 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:54.803316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.804249 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.804252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:54.804365 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:54.804394 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 22:40:54.804472 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 22:40:54.804562 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":28,"spinlock_wait_cycles":5248,"thread_start_us":137,"threads_started":1}
I20260925 22:40:54.804816 3839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:54.800266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:35683 (local address 127.2.74.195:45945)
0925 22:40:54.800483 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:54.800501 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:54.800616 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:54.800981 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:54.800984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:54.801026 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:54.801558 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:54.801564 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.802414 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.802417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:54.803547 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:54.803554 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.804393 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.804396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:54.804506 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:54.804552 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 22:40:54.804630 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 22:40:54.804698 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":28,"spinlock_wait_cycles":32128}
I20260925 22:40:54.804870 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:54.800340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:45945 (local address 127.2.74.194:35683)
0925 22:40:54.800487 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:54.800492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:54.800609 (+ 117us) server_negotiation.cc:407] Connection header received
0925 22:40:54.800767 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:54.800770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:54.800827 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:54.800916 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:54.801680 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.802297 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.803675 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.804281 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.804403 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:54.804437 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:54.804667 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 22:40:54.804739 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:40:54.805454 3760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c792a2cdc14d438b3b08abea585bd7" candidate_uuid: "56baba7fcf0c48f3994957d47649dc55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "022f7ddabcfc42c790f5c08e603c922b" is_pre_election: true
I20260925 22:40:54.805464 3620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c792a2cdc14d438b3b08abea585bd7" candidate_uuid: "56baba7fcf0c48f3994957d47649dc55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" is_pre_election: true
I20260925 22:40:54.805791 3760 raft_consensus.cc:2454] T 73c792a2cdc14d438b3b08abea585bd7 P 022f7ddabcfc42c790f5c08e603c922b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56baba7fcf0c48f3994957d47649dc55 in term 0.
I20260925 22:40:54.805800 3620 raft_consensus.cc:2454] T 73c792a2cdc14d438b3b08abea585bd7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56baba7fcf0c48f3994957d47649dc55 in term 0.
I20260925 22:40:54.806371 3831 leader_election.cc:302] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 022f7ddabcfc42c790f5c08e603c922b, 56baba7fcf0c48f3994957d47649dc55, d18a7a96e02f4f4fb960ffb0c8cbfbb0; no voters:
I20260925 22:40:54.806713 3977 raft_consensus.cc:2781] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:40:54.806794 3977 raft_consensus.cc:482] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:40:54.806825 3977 raft_consensus.cc:3037] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:54.807937 3977 raft_consensus.cc:504] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } }
I20260925 22:40:54.808374 3977 leader_election.cc:288] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55 [CANDIDATE]: Term 1 election: Requested vote from peers d18a7a96e02f4f4fb960ffb0c8cbfbb0 (127.2.74.193:40695), 022f7ddabcfc42c790f5c08e603c922b (127.2.74.194:35683)
I20260925 22:40:54.808820 3760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c792a2cdc14d438b3b08abea585bd7" candidate_uuid: "56baba7fcf0c48f3994957d47649dc55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "022f7ddabcfc42c790f5c08e603c922b"
I20260925 22:40:54.808899 3620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c792a2cdc14d438b3b08abea585bd7" candidate_uuid: "56baba7fcf0c48f3994957d47649dc55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0"
I20260925 22:40:54.808964 3760 raft_consensus.cc:3037] T 73c792a2cdc14d438b3b08abea585bd7 P 022f7ddabcfc42c790f5c08e603c922b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:54.809126 3620 raft_consensus.cc:3037] T 73c792a2cdc14d438b3b08abea585bd7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:54.810197 3620 raft_consensus.cc:2454] T 73c792a2cdc14d438b3b08abea585bd7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56baba7fcf0c48f3994957d47649dc55 in term 1.
I20260925 22:40:54.810196 3760 raft_consensus.cc:2454] T 73c792a2cdc14d438b3b08abea585bd7 P 022f7ddabcfc42c790f5c08e603c922b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56baba7fcf0c48f3994957d47649dc55 in term 1.
I20260925 22:40:54.810643 3831 leader_election.cc:302] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56baba7fcf0c48f3994957d47649dc55, d18a7a96e02f4f4fb960ffb0c8cbfbb0; no voters:
I20260925 22:40:54.810863 3977 raft_consensus.cc:2781] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:54.811131 3977 raft_consensus.cc:686] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55 [term 1 LEADER]: Becoming Leader. State: Replica: 56baba7fcf0c48f3994957d47649dc55, State: Running, Role: LEADER
I20260925 22:40:54.811524 3977 consensus_queue.cc:237] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } }
I20260925 22:40:54.814808 3463 catalog_manager.cc:5495] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56baba7fcf0c48f3994957d47649dc55 (127.2.74.195). New cstate: current_term: 1 leader_uuid: "56baba7fcf0c48f3994957d47649dc55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } health_report { overall_health: HEALTHY } } }
I20260925 22:40:54.878700 3968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:54.874513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50480 (local address 127.2.74.195:33851)
0925 22:40:54.874693 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:54.874699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:54.874715 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:40:54.874985 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:54.874988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:54.875057 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:54.875167 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:40:54.875460 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.875983 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.876978 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:54.877225 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:54.877338 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:54.877721 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:40:54.878194 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:40:54.878321 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:54.878453 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 22:40:54.878527 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 22:40:54.885106 3760 raft_consensus.cc:1261] T 73c792a2cdc14d438b3b08abea585bd7 P 022f7ddabcfc42c790f5c08e603c922b [term 1 FOLLOWER]: Refusing update from remote peer 56baba7fcf0c48f3994957d47649dc55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:40:54.885106 3620 raft_consensus.cc:1261] T 73c792a2cdc14d438b3b08abea585bd7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0 [term 1 FOLLOWER]: Refusing update from remote peer 56baba7fcf0c48f3994957d47649dc55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:40:54.885804 3979 consensus_queue.cc:1035] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:40:54.886029 3977 consensus_queue.cc:1035] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:40:54.893697 3982 mvcc.cc:205] Tried to move back new op lower bound from 7333380320801890304 to 7333380320510656512. Current Snapshot: MvccSnapshot[applied={T|T < 7333380320801890304}]
I20260925 22:40:54.906136 3463 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:40:54.906350 3463 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:40:54.908973 3463 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49e3aa4e21444fbcac571579b544ff97: Sending DeleteTablet for 3 replicas of tablet 73c792a2cdc14d438b3b08abea585bd7
I20260925 22:40:54.909849 3740 tablet_service.cc:1514] Processing DeleteTablet for tablet 73c792a2cdc14d438b3b08abea585bd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:54 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:38940
I20260925 22:40:54.909848 3600 tablet_service.cc:1514] Processing DeleteTablet for tablet 73c792a2cdc14d438b3b08abea585bd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:54 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:34358
I20260925 22:40:54.910694 3880 tablet_service.cc:1514] Processing DeleteTablet for tablet 73c792a2cdc14d438b3b08abea585bd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:54 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:50474
I20260925 22:40:54.910923 3463 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 22:40:54.911129 3993 tablet_replica.cc:330] T 73c792a2cdc14d438b3b08abea585bd7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0: stopping tablet replica
W20260925 22:40:54.911432 3463 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:40:54.911494 3993 raft_consensus.cc:2229] T 73c792a2cdc14d438b3b08abea585bd7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:40:54.911849 3993 raft_consensus.cc:2258] T 73c792a2cdc14d438b3b08abea585bd7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:40:54.912305 3992 tablet_replica.cc:330] T 73c792a2cdc14d438b3b08abea585bd7 P 022f7ddabcfc42c790f5c08e603c922b: stopping tablet replica
I20260925 22:40:54.912914 3994 tablet_replica.cc:330] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55: stopping tablet replica
I20260925 22:40:54.913208 3994 raft_consensus.cc:2229] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:40:54.913573 3994 raft_consensus.cc:2258] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:40:54.913612 3993 ts_tablet_manager.cc:1905] T 73c792a2cdc14d438b3b08abea585bd7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:40:54.914937 3994 ts_tablet_manager.cc:1905] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:40:54.916750 3993 ts_tablet_manager.cc:1918] T 73c792a2cdc14d438b3b08abea585bd7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:40:54.916940 3993 log.cc:1196] T 73c792a2cdc14d438b3b08abea585bd7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-0/wal/wals/73c792a2cdc14d438b3b08abea585bd7
I20260925 22:40:54.917349 3993 ts_tablet_manager.cc:1939] T 73c792a2cdc14d438b3b08abea585bd7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0: Deleting consensus metadata
I20260925 22:40:54.918184 3600 tablet_service.cc:1467] Processing CreateTablet for tablet c4b2d78beec7456bb8cbfc94e53b5da7 (DEFAULT_TABLE table=test-table [id=33db9e62fb10454184c4cb01a4f94162]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:54.918267 3994 ts_tablet_manager.cc:1918] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:40:54.918365 3994 log.cc:1196] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-2/wal/wals/73c792a2cdc14d438b3b08abea585bd7
I20260925 22:40:54.918454 3600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4b2d78beec7456bb8cbfc94e53b5da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:54.918684 3994 ts_tablet_manager.cc:1939] T 73c792a2cdc14d438b3b08abea585bd7 P 56baba7fcf0c48f3994957d47649dc55: Deleting consensus metadata
I20260925 22:40:54.919806 3447 catalog_manager.cc:4841] TS d18a7a96e02f4f4fb960ffb0c8cbfbb0 (127.2.74.193:40695): tablet 73c792a2cdc14d438b3b08abea585bd7 (table test-table [id=6cd3afaff39c4d2897fd2c510d33fb54]) successfully deleted
I20260925 22:40:54.920174 3880 tablet_service.cc:1467] Processing CreateTablet for tablet c4b2d78beec7456bb8cbfc94e53b5da7 (DEFAULT_TABLE table=test-table [id=33db9e62fb10454184c4cb01a4f94162]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:54.920437 3880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4b2d78beec7456bb8cbfc94e53b5da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:54.920713 3970 tablet_bootstrap.cc:492] T c4b2d78beec7456bb8cbfc94e53b5da7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0: Bootstrap starting.
I20260925 22:40:54.920758 3448 catalog_manager.cc:4841] TS 56baba7fcf0c48f3994957d47649dc55 (127.2.74.195:33851): tablet 73c792a2cdc14d438b3b08abea585bd7 (table test-table [id=6cd3afaff39c4d2897fd2c510d33fb54]) successfully deleted
I20260925 22:40:54.921713 3970 tablet_bootstrap.cc:654] T c4b2d78beec7456bb8cbfc94e53b5da7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:54.922788 3971 tablet_bootstrap.cc:492] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55: Bootstrap starting.
I20260925 22:40:54.923026 3970 tablet_bootstrap.cc:492] T c4b2d78beec7456bb8cbfc94e53b5da7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0: No bootstrap required, opened a new log
I20260925 22:40:54.923112 3970 ts_tablet_manager.cc:1397] T c4b2d78beec7456bb8cbfc94e53b5da7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:40:54.923031 3740 tablet_service.cc:1467] Processing CreateTablet for tablet c4b2d78beec7456bb8cbfc94e53b5da7 (DEFAULT_TABLE table=test-table [id=33db9e62fb10454184c4cb01a4f94162]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:54.923691 3740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4b2d78beec7456bb8cbfc94e53b5da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:54.923761 3971 tablet_bootstrap.cc:654] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:54.923800 3970 raft_consensus.cc:348] T c4b2d78beec7456bb8cbfc94e53b5da7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } }
I20260925 22:40:54.923903 3970 raft_consensus.cc:374] T c4b2d78beec7456bb8cbfc94e53b5da7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:54.923934 3970 raft_consensus.cc:729] T c4b2d78beec7456bb8cbfc94e53b5da7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d18a7a96e02f4f4fb960ffb0c8cbfbb0, State: Initialized, Role: FOLLOWER
I20260925 22:40:54.924069 3970 consensus_queue.cc:260] T c4b2d78beec7456bb8cbfc94e53b5da7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } }
I20260925 22:40:54.924321 3970 ts_tablet_manager.cc:1428] T c4b2d78beec7456bb8cbfc94e53b5da7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:40:54.912635 3992 raft_consensus.cc:2229] T 73c792a2cdc14d438b3b08abea585bd7 P 022f7ddabcfc42c790f5c08e603c922b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:40:54.925979 3969 tablet_bootstrap.cc:492] T c4b2d78beec7456bb8cbfc94e53b5da7 P 022f7ddabcfc42c790f5c08e603c922b: Bootstrap starting.
I20260925 22:40:54.926347 3992 raft_consensus.cc:2258] T 73c792a2cdc14d438b3b08abea585bd7 P 022f7ddabcfc42c790f5c08e603c922b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:40:54.927233 3969 tablet_bootstrap.cc:654] T c4b2d78beec7456bb8cbfc94e53b5da7 P 022f7ddabcfc42c790f5c08e603c922b: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:54.928153 3971 tablet_bootstrap.cc:492] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55: No bootstrap required, opened a new log
I20260925 22:40:54.928247 3971 ts_tablet_manager.cc:1397] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260925 22:40:54.928753 3971 raft_consensus.cc:348] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } }
I20260925 22:40:54.928862 3971 raft_consensus.cc:374] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:54.928890 3971 raft_consensus.cc:729] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56baba7fcf0c48f3994957d47649dc55, State: Initialized, Role: FOLLOWER
I20260925 22:40:54.929008 3971 consensus_queue.cc:260] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } }
I20260925 22:40:54.929044 3992 ts_tablet_manager.cc:1905] T 73c792a2cdc14d438b3b08abea585bd7 P 022f7ddabcfc42c790f5c08e603c922b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:40:54.929251 3971 ts_tablet_manager.cc:1428] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:40:54.931891 3992 ts_tablet_manager.cc:1918] T 73c792a2cdc14d438b3b08abea585bd7 P 022f7ddabcfc42c790f5c08e603c922b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:40:54.932011 3992 log.cc:1196] T 73c792a2cdc14d438b3b08abea585bd7 P 022f7ddabcfc42c790f5c08e603c922b: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-1/wal/wals/73c792a2cdc14d438b3b08abea585bd7
I20260925 22:40:54.932714 3992 ts_tablet_manager.cc:1939] T 73c792a2cdc14d438b3b08abea585bd7 P 022f7ddabcfc42c790f5c08e603c922b: Deleting consensus metadata
I20260925 22:40:54.933943 3450 catalog_manager.cc:4841] TS 022f7ddabcfc42c790f5c08e603c922b (127.2.74.194:35683): tablet 73c792a2cdc14d438b3b08abea585bd7 (table test-table [id=6cd3afaff39c4d2897fd2c510d33fb54]) successfully deleted
I20260925 22:40:54.934674 3969 tablet_bootstrap.cc:492] T c4b2d78beec7456bb8cbfc94e53b5da7 P 022f7ddabcfc42c790f5c08e603c922b: No bootstrap required, opened a new log
I20260925 22:40:54.935252 3969 ts_tablet_manager.cc:1397] T c4b2d78beec7456bb8cbfc94e53b5da7 P 022f7ddabcfc42c790f5c08e603c922b: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260925 22:40:54.935761 3969 raft_consensus.cc:348] T c4b2d78beec7456bb8cbfc94e53b5da7 P 022f7ddabcfc42c790f5c08e603c922b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } }
I20260925 22:40:54.935887 3969 raft_consensus.cc:374] T c4b2d78beec7456bb8cbfc94e53b5da7 P 022f7ddabcfc42c790f5c08e603c922b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:54.935920 3969 raft_consensus.cc:729] T c4b2d78beec7456bb8cbfc94e53b5da7 P 022f7ddabcfc42c790f5c08e603c922b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 022f7ddabcfc42c790f5c08e603c922b, State: Initialized, Role: FOLLOWER
I20260925 22:40:54.936057 3969 consensus_queue.cc:260] T c4b2d78beec7456bb8cbfc94e53b5da7 P 022f7ddabcfc42c790f5c08e603c922b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } }
I20260925 22:40:54.936352 3969 ts_tablet_manager.cc:1428] T c4b2d78beec7456bb8cbfc94e53b5da7 P 022f7ddabcfc42c790f5c08e603c922b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:40:55.026014 3989 raft_consensus.cc:482] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:40:55.026176 3989 raft_consensus.cc:504] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } }
I20260925 22:40:55.026696 3989 leader_election.cc:288] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d18a7a96e02f4f4fb960ffb0c8cbfbb0 (127.2.74.193:40695), 022f7ddabcfc42c790f5c08e603c922b (127.2.74.194:35683)
I20260925 22:40:55.027112 3620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b2d78beec7456bb8cbfc94e53b5da7" candidate_uuid: "56baba7fcf0c48f3994957d47649dc55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" is_pre_election: true
I20260925 22:40:55.027141 3760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b2d78beec7456bb8cbfc94e53b5da7" candidate_uuid: "56baba7fcf0c48f3994957d47649dc55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "022f7ddabcfc42c790f5c08e603c922b" is_pre_election: true
I20260925 22:40:55.027292 3620 raft_consensus.cc:2454] T c4b2d78beec7456bb8cbfc94e53b5da7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56baba7fcf0c48f3994957d47649dc55 in term 0.
I20260925 22:40:55.027310 3760 raft_consensus.cc:2454] T c4b2d78beec7456bb8cbfc94e53b5da7 P 022f7ddabcfc42c790f5c08e603c922b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56baba7fcf0c48f3994957d47649dc55 in term 0.
I20260925 22:40:55.027643 3831 leader_election.cc:302] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56baba7fcf0c48f3994957d47649dc55, d18a7a96e02f4f4fb960ffb0c8cbfbb0; no voters:
I20260925 22:40:55.027853 3989 raft_consensus.cc:2781] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:40:55.027936 3989 raft_consensus.cc:482] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:40:55.027990 3989 raft_consensus.cc:3037] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:55.029268 3989 raft_consensus.cc:504] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } }
I20260925 22:40:55.029675 3989 leader_election.cc:288] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55 [CANDIDATE]: Term 1 election: Requested vote from peers d18a7a96e02f4f4fb960ffb0c8cbfbb0 (127.2.74.193:40695), 022f7ddabcfc42c790f5c08e603c922b (127.2.74.194:35683)
I20260925 22:40:55.030146 3760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b2d78beec7456bb8cbfc94e53b5da7" candidate_uuid: "56baba7fcf0c48f3994957d47649dc55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "022f7ddabcfc42c790f5c08e603c922b"
I20260925 22:40:55.030301 3760 raft_consensus.cc:3037] T c4b2d78beec7456bb8cbfc94e53b5da7 P 022f7ddabcfc42c790f5c08e603c922b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:55.030315 3620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b2d78beec7456bb8cbfc94e53b5da7" candidate_uuid: "56baba7fcf0c48f3994957d47649dc55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0"
I20260925 22:40:55.030459 3620 raft_consensus.cc:3037] T c4b2d78beec7456bb8cbfc94e53b5da7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:55.031402 3760 raft_consensus.cc:2454] T c4b2d78beec7456bb8cbfc94e53b5da7 P 022f7ddabcfc42c790f5c08e603c922b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56baba7fcf0c48f3994957d47649dc55 in term 1.
I20260925 22:40:55.031716 3620 raft_consensus.cc:2454] T c4b2d78beec7456bb8cbfc94e53b5da7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56baba7fcf0c48f3994957d47649dc55 in term 1.
I20260925 22:40:55.031786 3834 leader_election.cc:302] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 022f7ddabcfc42c790f5c08e603c922b, 56baba7fcf0c48f3994957d47649dc55; no voters:
I20260925 22:40:55.032174 3989 raft_consensus.cc:2781] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:55.032279 3989 raft_consensus.cc:686] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55 [term 1 LEADER]: Becoming Leader. State: Replica: 56baba7fcf0c48f3994957d47649dc55, State: Running, Role: LEADER
I20260925 22:40:55.032431 3989 consensus_queue.cc:237] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } }
I20260925 22:40:55.034487 3463 catalog_manager.cc:5495] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56baba7fcf0c48f3994957d47649dc55 (127.2.74.195). New cstate: current_term: 1 leader_uuid: "56baba7fcf0c48f3994957d47649dc55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } health_report { overall_health: HEALTHY } } }
Sep 25 22:40:55 dist-test-slave-csnl krb5kdc[3418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376053, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:40:55.085458 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:55.072799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:38813 (local address 127.0.0.1:52244)
0925 22:40:55.073115 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:55.073135 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:55.073245 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:55.073535 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:55.073537 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:55.073922 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:55.074023 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:55.074029 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:55.074788 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:55.074790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:55.075388 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:55.075396 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:55.075894 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:55.075897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:55.075959 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:55.080524 (+ 4565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:55.080538 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:55.082114 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:55.083895 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:55.083904 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:55.083910 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:55.084160 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:55.084408 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:55.084410 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:55.084411 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:55.084487 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:55.084841 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:55.084846 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:55.084952 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 22:40:55.085044 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 22:40:55.085230 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":124,"threads_started":1}
I20260925 22:40:55.085464 4005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:55.072887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52244 (local address 127.2.74.254:38813)
0925 22:40:55.073185 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:55.073195 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:55.073217 (+ 22us) server_negotiation.cc:407] Connection header received
0925 22:40:55.073347 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:55.073351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:55.073406 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:55.073497 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:55.074157 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:55.074690 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:55.075484 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:55.075771 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:55.075869 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:55.082310 (+ 6441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:55.082313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:55.082315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:55.082337 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:55.083769 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:55.084250 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:55.084253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:55.084254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:55.084330 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:55.084562 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:55.084565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:55.084566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:55.084751 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:55.084827 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:55.085137 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 22:40:55.085294 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":226,"thread_start_us":106,"threads_started":1}
I20260925 22:40:55.089433 3463 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:52244:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 22:40:55.090232 3463 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:40:55.095523 3740 tablet_service.cc:1467] Processing CreateTablet for tablet 1f3480e8f0834e25888041700cdf298d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4b4576c6c0f942c3b0ad8184bd57801b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:40:55.095468 3600 tablet_service.cc:1467] Processing CreateTablet for tablet 1f3480e8f0834e25888041700cdf298d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4b4576c6c0f942c3b0ad8184bd57801b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:40:55.095587 3880 tablet_service.cc:1467] Processing CreateTablet for tablet 1f3480e8f0834e25888041700cdf298d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4b4576c6c0f942c3b0ad8184bd57801b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:40:55.095881 3740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f3480e8f0834e25888041700cdf298d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:55.095884 3600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f3480e8f0834e25888041700cdf298d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:55.095994 3880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f3480e8f0834e25888041700cdf298d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:55.098361 3971 tablet_bootstrap.cc:492] T 1f3480e8f0834e25888041700cdf298d P 56baba7fcf0c48f3994957d47649dc55: Bootstrap starting.
I20260925 22:40:55.098838 3969 tablet_bootstrap.cc:492] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b: Bootstrap starting.
I20260925 22:40:55.099386 3971 tablet_bootstrap.cc:654] T 1f3480e8f0834e25888041700cdf298d P 56baba7fcf0c48f3994957d47649dc55: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:55.099826 3969 tablet_bootstrap.cc:654] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:55.100304 3970 tablet_bootstrap.cc:492] T 1f3480e8f0834e25888041700cdf298d P d18a7a96e02f4f4fb960ffb0c8cbfbb0: Bootstrap starting.
I20260925 22:40:55.101258 3970 tablet_bootstrap.cc:654] T 1f3480e8f0834e25888041700cdf298d P d18a7a96e02f4f4fb960ffb0c8cbfbb0: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:55.101682 3969 tablet_bootstrap.cc:492] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b: No bootstrap required, opened a new log
I20260925 22:40:55.101783 3969 ts_tablet_manager.cc:1397] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:40:55.102290 3971 tablet_bootstrap.cc:492] T 1f3480e8f0834e25888041700cdf298d P 56baba7fcf0c48f3994957d47649dc55: No bootstrap required, opened a new log
I20260925 22:40:55.102257 3969 raft_consensus.cc:348] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } }
I20260925 22:40:55.102358 3969 raft_consensus.cc:374] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:55.102406 3971 ts_tablet_manager.cc:1397] T 1f3480e8f0834e25888041700cdf298d P 56baba7fcf0c48f3994957d47649dc55: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 22:40:55.102417 3969 raft_consensus.cc:729] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 022f7ddabcfc42c790f5c08e603c922b, State: Initialized, Role: FOLLOWER
I20260925 22:40:55.102551 3969 consensus_queue.cc:260] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } }
I20260925 22:40:55.102854 3969 ts_tablet_manager.cc:1428] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:40:55.102814 3976 tablet_replica.cc:439] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } } }
I20260925 22:40:55.102989 3976 tablet_replica.cc:442] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:40:55.102885 3971 raft_consensus.cc:348] T 1f3480e8f0834e25888041700cdf298d P 56baba7fcf0c48f3994957d47649dc55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } }
I20260925 22:40:55.103061 3971 raft_consensus.cc:374] T 1f3480e8f0834e25888041700cdf298d P 56baba7fcf0c48f3994957d47649dc55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:55.103129 3971 raft_consensus.cc:729] T 1f3480e8f0834e25888041700cdf298d P 56baba7fcf0c48f3994957d47649dc55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56baba7fcf0c48f3994957d47649dc55, State: Initialized, Role: FOLLOWER
I20260925 22:40:55.103281 3971 consensus_queue.cc:260] T 1f3480e8f0834e25888041700cdf298d P 56baba7fcf0c48f3994957d47649dc55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } }
I20260925 22:40:55.103566 3971 ts_tablet_manager.cc:1428] T 1f3480e8f0834e25888041700cdf298d P 56baba7fcf0c48f3994957d47649dc55: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:40:55.103603 3989 tablet_replica.cc:439] T 1f3480e8f0834e25888041700cdf298d P 56baba7fcf0c48f3994957d47649dc55: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } } }
I20260925 22:40:55.103755 3989 tablet_replica.cc:442] T 1f3480e8f0834e25888041700cdf298d P 56baba7fcf0c48f3994957d47649dc55: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:40:55.104692 3970 tablet_bootstrap.cc:492] T 1f3480e8f0834e25888041700cdf298d P d18a7a96e02f4f4fb960ffb0c8cbfbb0: No bootstrap required, opened a new log
I20260925 22:40:55.104796 3970 ts_tablet_manager.cc:1397] T 1f3480e8f0834e25888041700cdf298d P d18a7a96e02f4f4fb960ffb0c8cbfbb0: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 22:40:55.105274 3970 raft_consensus.cc:348] T 1f3480e8f0834e25888041700cdf298d P d18a7a96e02f4f4fb960ffb0c8cbfbb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } }
I20260925 22:40:55.105403 3970 raft_consensus.cc:374] T 1f3480e8f0834e25888041700cdf298d P d18a7a96e02f4f4fb960ffb0c8cbfbb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:55.105439 3970 raft_consensus.cc:729] T 1f3480e8f0834e25888041700cdf298d P d18a7a96e02f4f4fb960ffb0c8cbfbb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d18a7a96e02f4f4fb960ffb0c8cbfbb0, State: Initialized, Role: FOLLOWER
I20260925 22:40:55.105589 3970 consensus_queue.cc:260] T 1f3480e8f0834e25888041700cdf298d P d18a7a96e02f4f4fb960ffb0c8cbfbb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } }
I20260925 22:40:55.105927 3970 ts_tablet_manager.cc:1428] T 1f3480e8f0834e25888041700cdf298d P d18a7a96e02f4f4fb960ffb0c8cbfbb0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:40:55.105857 3975 tablet_replica.cc:439] T 1f3480e8f0834e25888041700cdf298d P d18a7a96e02f4f4fb960ffb0c8cbfbb0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } } }
I20260925 22:40:55.106009 3975 tablet_replica.cc:442] T 1f3480e8f0834e25888041700cdf298d P d18a7a96e02f4f4fb960ffb0c8cbfbb0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:40:55.207229 3976 raft_consensus.cc:482] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:40:55.207423 3976 raft_consensus.cc:504] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } }
I20260925 22:40:55.208664 3976 leader_election.cc:288] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d18a7a96e02f4f4fb960ffb0c8cbfbb0 (127.2.74.193:40695), 56baba7fcf0c48f3994957d47649dc55 (127.2.74.195:33851)
I20260925 22:40:55.214152 3964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:55.208976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:47351 (local address 127.2.74.193:40695)
0925 22:40:55.209116 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:55.209121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:55.209431 (+ 310us) server_negotiation.cc:407] Connection header received
0925 22:40:55.209563 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:55.209566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:55.209621 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:55.209707 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:55.210099 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:55.210871 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:55.212761 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:55.213406 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:55.213547 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:55.213605 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:55.213934 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 22:40:55.214009 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:40:55.214152 3968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:55.209044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:59823 (local address 127.2.74.195:33851)
0925 22:40:55.209175 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:55.209180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:55.209452 (+ 272us) server_negotiation.cc:407] Connection header received
0925 22:40:55.209562 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:55.209566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:55.209621 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:55.209707 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:55.210086 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:55.210871 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:55.212761 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:55.213406 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:55.213547 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:55.213605 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:55.213934 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 22:40:55.214010 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:40:55.214205 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:55.208877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:40695 (local address 127.2.74.194:47351)
0925 22:40:55.209299 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:55.209315 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:55.209451 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:55.209740 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:55.209743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:55.209781 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:55.209966 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:55.209972 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:55.211028 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:55.211033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:55.212588 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:55.212604 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:55.213594 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:55.213598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:55.213763 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:55.213804 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 22:40:55.213900 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 22:40:55.214001 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":218,"spinlock_wait_cycles":10240,"thread_start_us":137,"threads_started":1}
I20260925 22:40:55.214207 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:55.208953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:33851 (local address 127.2.74.194:59823)
0925 22:40:55.209355 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:55.209368 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:55.209451 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:55.209740 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:55.209743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:55.209780 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:55.209966 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:55.209972 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:55.211007 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:55.211014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:55.212589 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:55.212604 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:55.213594 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:55.213598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:55.213759 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:55.213813 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 22:40:55.213900 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 22:40:55.214004 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":208,"spinlock_wait_cycles":28416,"thread_start_us":131,"threads_started":1}
I20260925 22:40:55.214872 3620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f3480e8f0834e25888041700cdf298d" candidate_uuid: "022f7ddabcfc42c790f5c08e603c922b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" is_pre_election: true
I20260925 22:40:55.215096 3620 raft_consensus.cc:2454] T 1f3480e8f0834e25888041700cdf298d P d18a7a96e02f4f4fb960ffb0c8cbfbb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 022f7ddabcfc42c790f5c08e603c922b in term 0.
I20260925 22:40:55.215039 3900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f3480e8f0834e25888041700cdf298d" candidate_uuid: "022f7ddabcfc42c790f5c08e603c922b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56baba7fcf0c48f3994957d47649dc55" is_pre_election: true
I20260925 22:40:55.215282 3900 raft_consensus.cc:2454] T 1f3480e8f0834e25888041700cdf298d P 56baba7fcf0c48f3994957d47649dc55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 022f7ddabcfc42c790f5c08e603c922b in term 0.
I20260925 22:40:55.215555 3691 leader_election.cc:302] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 022f7ddabcfc42c790f5c08e603c922b, d18a7a96e02f4f4fb960ffb0c8cbfbb0; no voters:
I20260925 22:40:55.215852 3976 raft_consensus.cc:2781] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:40:55.215915 3976 raft_consensus.cc:482] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:40:55.215965 3976 raft_consensus.cc:3037] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:55.217329 3976 raft_consensus.cc:504] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } }
I20260925 22:40:55.217764 3976 leader_election.cc:288] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b [CANDIDATE]: Term 1 election: Requested vote from peers d18a7a96e02f4f4fb960ffb0c8cbfbb0 (127.2.74.193:40695), 56baba7fcf0c48f3994957d47649dc55 (127.2.74.195:33851)
I20260925 22:40:55.218189 3900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f3480e8f0834e25888041700cdf298d" candidate_uuid: "022f7ddabcfc42c790f5c08e603c922b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56baba7fcf0c48f3994957d47649dc55"
I20260925 22:40:55.218186 3620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f3480e8f0834e25888041700cdf298d" candidate_uuid: "022f7ddabcfc42c790f5c08e603c922b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0"
I20260925 22:40:55.218335 3900 raft_consensus.cc:3037] T 1f3480e8f0834e25888041700cdf298d P 56baba7fcf0c48f3994957d47649dc55 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:55.218335 3620 raft_consensus.cc:3037] T 1f3480e8f0834e25888041700cdf298d P d18a7a96e02f4f4fb960ffb0c8cbfbb0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:55.219569 3900 raft_consensus.cc:2454] T 1f3480e8f0834e25888041700cdf298d P 56baba7fcf0c48f3994957d47649dc55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 022f7ddabcfc42c790f5c08e603c922b in term 1.
I20260925 22:40:55.219569 3620 raft_consensus.cc:2454] T 1f3480e8f0834e25888041700cdf298d P d18a7a96e02f4f4fb960ffb0c8cbfbb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 022f7ddabcfc42c790f5c08e603c922b in term 1.
I20260925 22:40:55.219947 3691 leader_election.cc:302] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 022f7ddabcfc42c790f5c08e603c922b, d18a7a96e02f4f4fb960ffb0c8cbfbb0; no voters:
I20260925 22:40:55.220171 3976 raft_consensus.cc:2781] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:55.220420 3976 raft_consensus.cc:686] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b [term 1 LEADER]: Becoming Leader. State: Replica: 022f7ddabcfc42c790f5c08e603c922b, State: Running, Role: LEADER
I20260925 22:40:55.220715 3976 consensus_queue.cc:237] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } }
I20260925 22:40:55.221927 4011 tablet_replica.cc:439] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b: TxnStatusTablet state changed. Reason: New leader 022f7ddabcfc42c790f5c08e603c922b. Latest consensus state: current_term: 1 leader_uuid: "022f7ddabcfc42c790f5c08e603c922b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } } }
I20260925 22:40:55.222069 4011 tablet_replica.cc:442] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:40:55.222347 4013 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 22:40:55.222506 4013 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 22:40:55.223510 3462 catalog_manager.cc:5495] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b reported cstate change: term changed from 0 to 1, leader changed from <none> to 022f7ddabcfc42c790f5c08e603c922b (127.2.74.194). New cstate: current_term: 1 leader_uuid: "022f7ddabcfc42c790f5c08e603c922b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } health_report { overall_health: UNKNOWN } } }
I20260925 22:40:55.263150 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:55.259050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38954 (local address 127.2.74.194:35683)
0925 22:40:55.259192 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:55.259197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:55.259348 (+ 151us) server_negotiation.cc:407] Connection header received
0925 22:40:55.259612 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:55.259615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:55.259708 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:55.259820 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:40:55.260082 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:55.260656 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:55.261554 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:55.261797 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:55.261910 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:55.262246 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:40:55.262695 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:40:55.262822 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:55.262907 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 22:40:55.262994 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:40:55.263150 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:55.258976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:35683 (local address 127.0.0.1:38954)
0925 22:40:55.259257 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:55.259272 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:55.259416 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:55.259844 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:55.259847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:55.259856 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:40:55.259966 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:55.259979 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:55.260778 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:55.260781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:55.261418 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:55.261427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:55.261919 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:55.261922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:55.262017 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:55.262078 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:40:55.262799 (+ 721us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:40:55.262817 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 22:40:55.262887 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 22:40:55.263016 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260925 22:40:55.266544 3620 raft_consensus.cc:1261] T 1f3480e8f0834e25888041700cdf298d P d18a7a96e02f4f4fb960ffb0c8cbfbb0 [term 1 FOLLOWER]: Refusing update from remote peer 022f7ddabcfc42c790f5c08e603c922b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:40:55.266723 3900 raft_consensus.cc:1261] T 1f3480e8f0834e25888041700cdf298d P 56baba7fcf0c48f3994957d47649dc55 [term 1 FOLLOWER]: Refusing update from remote peer 022f7ddabcfc42c790f5c08e603c922b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:40:55.267190 4011 consensus_queue.cc:1035] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:40:55.267364 3976 consensus_queue.cc:1035] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b [LEADER]: Connected to new peer: Peer: permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:40:55.269269 3989 tablet_replica.cc:439] T 1f3480e8f0834e25888041700cdf298d P 56baba7fcf0c48f3994957d47649dc55: TxnStatusTablet state changed. Reason: New leader 022f7ddabcfc42c790f5c08e603c922b. Latest consensus state: current_term: 1 leader_uuid: "022f7ddabcfc42c790f5c08e603c922b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } } }
I20260925 22:40:55.269410 3989 tablet_replica.cc:442] T 1f3480e8f0834e25888041700cdf298d P 56baba7fcf0c48f3994957d47649dc55: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:40:55.270510 3975 tablet_replica.cc:439] T 1f3480e8f0834e25888041700cdf298d P d18a7a96e02f4f4fb960ffb0c8cbfbb0: TxnStatusTablet state changed. Reason: New leader 022f7ddabcfc42c790f5c08e603c922b. Latest consensus state: current_term: 1 leader_uuid: "022f7ddabcfc42c790f5c08e603c922b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } } }
I20260925 22:40:55.270668 3975 tablet_replica.cc:442] T 1f3480e8f0834e25888041700cdf298d P d18a7a96e02f4f4fb960ffb0c8cbfbb0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:40:55.271420 4011 tablet_replica.cc:439] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "022f7ddabcfc42c790f5c08e603c922b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } } }
I20260925 22:40:55.271539 4011 tablet_replica.cc:442] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:40:55.271652 3983 mvcc.cc:205] Tried to move back new op lower bound from 7333380322366160896 to 7333380322186657792. Current Snapshot: MvccSnapshot[applied={T|T < 7333380322366160896}]
I20260925 22:40:55.272033 3976 tablet_replica.cc:439] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "022f7ddabcfc42c790f5c08e603c922b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } } }
I20260925 22:40:55.272145 3976 tablet_replica.cc:442] T 1f3480e8f0834e25888041700cdf298d P 022f7ddabcfc42c790f5c08e603c922b: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:40:55.272817 3984 mvcc.cc:205] Tried to move back new op lower bound from 7333380322366160896 to 7333380322186657792. Current Snapshot: MvccSnapshot[applied={T|T < 7333380322366160896}]
I20260925 22:40:55.272974 3975 tablet_replica.cc:439] T 1f3480e8f0834e25888041700cdf298d P d18a7a96e02f4f4fb960ffb0c8cbfbb0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "022f7ddabcfc42c790f5c08e603c922b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } } }
I20260925 22:40:55.273092 3975 tablet_replica.cc:442] T 1f3480e8f0834e25888041700cdf298d P d18a7a96e02f4f4fb960ffb0c8cbfbb0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:40:55.273384 3989 tablet_replica.cc:439] T 1f3480e8f0834e25888041700cdf298d P 56baba7fcf0c48f3994957d47649dc55: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "022f7ddabcfc42c790f5c08e603c922b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 } } peers { permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 } } peers { permanent_uuid: "56baba7fcf0c48f3994957d47649dc55" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33851 } } }
I20260925 22:40:55.273497 3989 tablet_replica.cc:442] T 1f3480e8f0834e25888041700cdf298d P 56baba7fcf0c48f3994957d47649dc55: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:40:55.289054 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:55.285153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:49199 (local address 127.2.74.194:35683)
0925 22:40:55.285331 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:55.285338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:55.285532 (+ 194us) server_negotiation.cc:407] Connection header received
0925 22:40:55.285708 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:55.285710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:55.285762 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:55.285829 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:40:55.286075 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:55.286679 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:55.287969 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:55.288186 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:55.288274 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:55.288621 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:40:55.288696 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:40:55.288749 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:55.288879 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 22:40:55.288928 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 22:40:55.289124 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:55.285065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:35683 (local address 127.2.74.195:49199)
0925 22:40:55.285424 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:55.285443 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:55.285587 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:55.285847 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:55.285850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:55.285858 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:40:55.285985 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:55.285991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:55.286783 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:55.286787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:55.287833 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:55.287845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:55.288315 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:55.288319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:55.288412 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:55.288474 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:40:55.288780 (+ 306us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:40:55.288787 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 22:40:55.288862 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 22:40:55.288960 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":123,"threads_started":1}
I20260925 22:40:55.296629 3760 raft_consensus.cc:1261] T c4b2d78beec7456bb8cbfc94e53b5da7 P 022f7ddabcfc42c790f5c08e603c922b [term 1 FOLLOWER]: Refusing update from remote peer 56baba7fcf0c48f3994957d47649dc55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:40:55.296870 3620 raft_consensus.cc:1261] T c4b2d78beec7456bb8cbfc94e53b5da7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0 [term 1 FOLLOWER]: Refusing update from remote peer 56baba7fcf0c48f3994957d47649dc55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:40:55.297259 3989 consensus_queue.cc:1035] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d18a7a96e02f4f4fb960ffb0c8cbfbb0" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:40:55.297483 3977 consensus_queue.cc:1035] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "022f7ddabcfc42c790f5c08e603c922b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 35683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:40:55.323987 3968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:55.320106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:53579 (local address 127.2.74.195:33851)
0925 22:40:55.320230 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:55.320234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:55.320532 (+ 298us) server_negotiation.cc:407] Connection header received
0925 22:40:55.320647 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:55.320651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:55.320710 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:55.320787 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:40:55.321083 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:55.321876 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:55.322729 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:55.323103 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:55.323211 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:55.323466 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:40:55.323560 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:40:55.323628 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:55.323772 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 22:40:55.323836 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:40:55.324394 4025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:55.319974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:33851 (local address 127.2.74.194:53579)
0925 22:40:55.320431 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:55.320446 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:55.320559 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:55.320825 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:55.320830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:55.320838 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:40:55.320959 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:55.320964 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:55.321979 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:55.321981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:55.322608 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:55.322617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:55.323218 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:55.323220 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:55.323285 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:55.323336 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:40:55.323670 (+ 334us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:40:55.323676 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:40:55.324147 (+ 471us) client_negotiation.cc:240] Negotiation successful
0925 22:40:55.324240 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":122,"threads_started":1}
I20260925 22:40:55.353201 3462 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:40:55.353440 3462 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:40:55.355881 3462 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49e3aa4e21444fbcac571579b544ff97: Sending DeleteTablet for 3 replicas of tablet c4b2d78beec7456bb8cbfc94e53b5da7
I20260925 22:40:55.356556 3600 tablet_service.cc:1514] Processing DeleteTablet for tablet c4b2d78beec7456bb8cbfc94e53b5da7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:55 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:34358
I20260925 22:40:55.356640 3740 tablet_service.cc:1514] Processing DeleteTablet for tablet c4b2d78beec7456bb8cbfc94e53b5da7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:55 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:38940
I20260925 22:40:55.356839 3992 tablet_replica.cc:330] T c4b2d78beec7456bb8cbfc94e53b5da7 P 022f7ddabcfc42c790f5c08e603c922b: stopping tablet replica
I20260925 22:40:55.356830 3880 tablet_service.cc:1514] Processing DeleteTablet for tablet c4b2d78beec7456bb8cbfc94e53b5da7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:55 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:50474
I20260925 22:40:55.356863 3993 tablet_replica.cc:330] T c4b2d78beec7456bb8cbfc94e53b5da7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0: stopping tablet replica
I20260925 22:40:55.356964 3993 raft_consensus.cc:2229] T c4b2d78beec7456bb8cbfc94e53b5da7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:40:55.356964 3992 raft_consensus.cc:2229] T c4b2d78beec7456bb8cbfc94e53b5da7 P 022f7ddabcfc42c790f5c08e603c922b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:40:55.357107 3992 raft_consensus.cc:2258] T c4b2d78beec7456bb8cbfc94e53b5da7 P 022f7ddabcfc42c790f5c08e603c922b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:40:55.357107 3993 raft_consensus.cc:2258] T c4b2d78beec7456bb8cbfc94e53b5da7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:40:55.357304 3994 tablet_replica.cc:330] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55: stopping tablet replica
I20260925 22:40:55.357429 3994 raft_consensus.cc:2229] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:40:55.357641 3994 raft_consensus.cc:2258] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:40:55.357944 3993 ts_tablet_manager.cc:1905] T c4b2d78beec7456bb8cbfc94e53b5da7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:40:55.358163 3992 ts_tablet_manager.cc:1905] T c4b2d78beec7456bb8cbfc94e53b5da7 P 022f7ddabcfc42c790f5c08e603c922b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:40:55.359171 3994 ts_tablet_manager.cc:1905] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:40:55.360986 3992 ts_tablet_manager.cc:1918] T c4b2d78beec7456bb8cbfc94e53b5da7 P 022f7ddabcfc42c790f5c08e603c922b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:40:55.360981 3993 ts_tablet_manager.cc:1918] T c4b2d78beec7456bb8cbfc94e53b5da7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:40:55.361060 3993 log.cc:1196] T c4b2d78beec7456bb8cbfc94e53b5da7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-0/wal/wals/c4b2d78beec7456bb8cbfc94e53b5da7
I20260925 22:40:55.361075 3992 log.cc:1196] T c4b2d78beec7456bb8cbfc94e53b5da7 P 022f7ddabcfc42c790f5c08e603c922b: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-1/wal/wals/c4b2d78beec7456bb8cbfc94e53b5da7
I20260925 22:40:55.361337 3993 ts_tablet_manager.cc:1939] T c4b2d78beec7456bb8cbfc94e53b5da7 P d18a7a96e02f4f4fb960ffb0c8cbfbb0: Deleting consensus metadata
I20260925 22:40:55.361336 3992 ts_tablet_manager.cc:1939] T c4b2d78beec7456bb8cbfc94e53b5da7 P 022f7ddabcfc42c790f5c08e603c922b: Deleting consensus metadata
I20260925 22:40:55.361632 3994 ts_tablet_manager.cc:1918] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:40:55.361690 3994 log.cc:1196] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376047580784-2347-0/minicluster-data/ts-2/wal/wals/c4b2d78beec7456bb8cbfc94e53b5da7
I20260925 22:40:55.361877 3994 ts_tablet_manager.cc:1939] T c4b2d78beec7456bb8cbfc94e53b5da7 P 56baba7fcf0c48f3994957d47649dc55: Deleting consensus metadata
I20260925 22:40:55.362224 3450 catalog_manager.cc:4841] TS 022f7ddabcfc42c790f5c08e603c922b (127.2.74.194:35683): tablet c4b2d78beec7456bb8cbfc94e53b5da7 (table test-table [id=33db9e62fb10454184c4cb01a4f94162]) successfully deleted
I20260925 22:40:55.362903 3447 catalog_manager.cc:4841] TS d18a7a96e02f4f4fb960ffb0c8cbfbb0 (127.2.74.193:40695): tablet c4b2d78beec7456bb8cbfc94e53b5da7 (table test-table [id=33db9e62fb10454184c4cb01a4f94162]) successfully deleted
I20260925 22:40:55.363314 3448 catalog_manager.cc:4841] TS 56baba7fcf0c48f3994957d47649dc55 (127.2.74.195:33851): tablet c4b2d78beec7456bb8cbfc94e53b5da7 (table test-table [id=33db9e62fb10454184c4cb01a4f94162]) successfully deleted
Sep 25 22:40:55 dist-test-slave-csnl krb5kdc[3418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376054, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.74.193@KRBTEST.COM
I20260925 22:40:55.372838 3964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:55.362360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34370 (local address 127.2.74.193:40695)
0925 22:40:55.362568 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:55.362572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:55.362582 (+ 10us) server_negotiation.cc:407] Connection header received
0925 22:40:55.362612 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:55.362614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:55.362652 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:55.362752 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:55.363705 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:55.364226 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:55.365106 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:55.365343 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:55.365456 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:55.368781 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:55.368803 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:55.368818 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:55.368849 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:55.370723 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:55.371329 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:55.371336 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:55.371337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:55.371426 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:55.371762 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:55.371765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:55.371767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:55.372064 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:55.372309 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:55.372469 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 22:40:55.372598 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
W20260925 22:40:55.373301 3660 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34370
I20260925 22:40:55.385093 4005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:55.375224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52256 (local address 127.2.74.254:38813)
0925 22:40:55.375385 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:55.375389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:55.375474 (+ 85us) server_negotiation.cc:407] Connection header received
0925 22:40:55.375646 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:55.375649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:55.375730 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:55.375811 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:55.377339 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:55.377882 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:55.379127 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:55.379478 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:55.379569 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:55.381416 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:55.381421 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:55.381424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:55.381461 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:55.383307 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:55.383795 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:55.383799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:55.383801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:55.383905 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:55.384168 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:55.384171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:55.384173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:55.384405 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:55.384547 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:55.384764 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 22:40:55.384921 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 22:40:55.385603 3462 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52256
I20260925 22:40:55.386881 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 3535
I20260925 22:40:55.394526 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 3675
I20260925 22:40:55.401677 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 3815
I20260925 22:40:55.409132 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 3431
2026-09-25T22:40:55Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3166 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:40:55 dist-test-slave-csnl krb5kdc[4033](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:40:55 dist-test-slave-csnl krb5kdc[4033](info): set up 2 sockets
Sep 25 22:40:55 dist-test-slave-csnl krb5kdc[4033](info): commencing operation
krb5kdc: starting...
W20260925 22:40:56.443095 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:40:56 dist-test-slave-csnl krb5kdc[4033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:40:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:40:56.566277 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:33183
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:45851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:33183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:56.682703 4046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:56.683106 4046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:56.683224 4046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:56.687541 4046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:56.687667 4046 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 22:40:56.687697 4046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:56.687741 4046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:56.687778 4046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:56.687809 4046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:56.693611 4046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:45851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:33183
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:33183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:56.694852 4046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:56.695986 4046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:56.705098 4051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:56.705137 4054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:56.705109 4052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:56.705192 4046 server_base.cc:1034] running on GCE node
I20260925 22:40:56.706113 4046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:56.707572 4046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:56.708781 4046 hybrid_clock.cc:648] HybridClock initialized: now 1790376056708761 us; error 41 us; skew 500 ppm
Sep 25 22:40:56 dist-test-slave-csnl krb5kdc[4033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376056, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:56.711735 4046 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:40:56.712803 4046 webserver.cc:460] Webserver started at http://127.2.74.254:41217/ using document root <none> and password file <none>
I20260925 22:40:56.713433 4046 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:56.713533 4046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:56.713784 4046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:56.716110 4046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "b92fc40cacb846a1990f43fa527acdcb"
format_stamp: "Formatted at 2026-09-25 22:40:56 on dist-test-slave-csnl"
server_key: "e57c15d37fbcaa0234037d76380164bc"
server_key_iv: "5d6a71cd6cb8e9cf965ab3a529d88a3c"
server_key_version: "encryptionkey@0"
I20260925 22:40:56.716763 4046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "b92fc40cacb846a1990f43fa527acdcb"
format_stamp: "Formatted at 2026-09-25 22:40:56 on dist-test-slave-csnl"
server_key: "e57c15d37fbcaa0234037d76380164bc"
server_key_iv: "5d6a71cd6cb8e9cf965ab3a529d88a3c"
server_key_version: "encryptionkey@0"
I20260925 22:40:56.721232 4046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:40:56.724327 4061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:56.725800 4046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:40:56.726006 4046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "b92fc40cacb846a1990f43fa527acdcb"
format_stamp: "Formatted at 2026-09-25 22:40:56 on dist-test-slave-csnl"
server_key: "e57c15d37fbcaa0234037d76380164bc"
server_key_iv: "5d6a71cd6cb8e9cf965ab3a529d88a3c"
server_key_version: "encryptionkey@0"
I20260925 22:40:56.726162 4046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:56.748049 4046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:56.751372 4046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:56.751616 4046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:56.760040 4046 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:33183
I20260925 22:40:56.760067 4123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:33183 every 8 connection(s)
I20260925 22:40:56.761137 4046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:56.763399 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 4046
I20260925 22:40:56.763499 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:40:56.763783 2347 external_mini_cluster.cc:1428] Setting key cf563ff9559680281e29575c122b4e96
I20260925 22:40:56.764340 4124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:56.772059 4124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b92fc40cacb846a1990f43fa527acdcb: Bootstrap starting.
Sep 25 22:40:56 dist-test-slave-csnl krb5kdc[4033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:40:56.775089 4124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b92fc40cacb846a1990f43fa527acdcb: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:56.776158 4124 log.cc:824] T 00000000000000000000000000000000 P b92fc40cacb846a1990f43fa527acdcb: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:56.778918 4124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b92fc40cacb846a1990f43fa527acdcb: No bootstrap required, opened a new log
I20260925 22:40:56.780903 4127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:56.765355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55456 (local address 127.2.74.254:33183)
0925 22:40:56.765806 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:56.765827 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:56.765870 (+ 43us) server_negotiation.cc:407] Connection header received
0925 22:40:56.766700 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:56.766726 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:56.767205 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:56.767599 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:56.768258 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:56.769784 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:56.770738 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:56.771503 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:56.771677 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:56.774573 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:56.774581 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:56.774600 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:56.774641 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:56.777435 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:56.778017 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:56.778024 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:56.778030 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:56.778192 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:56.778482 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:56.778485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:56.778487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:56.779076 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:56.779271 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:56.779562 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 22:40:56.779889 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":120,"threads_started":1}
I20260925 22:40:56.783151 4124 raft_consensus.cc:348] T 00000000000000000000000000000000 P b92fc40cacb846a1990f43fa527acdcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92fc40cacb846a1990f43fa527acdcb" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33183 } }
I20260925 22:40:56.783481 4124 raft_consensus.cc:374] T 00000000000000000000000000000000 P b92fc40cacb846a1990f43fa527acdcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:56.783587 4124 raft_consensus.cc:729] T 00000000000000000000000000000000 P b92fc40cacb846a1990f43fa527acdcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92fc40cacb846a1990f43fa527acdcb, State: Initialized, Role: FOLLOWER
I20260925 22:40:56.784041 4124 consensus_queue.cc:260] T 00000000000000000000000000000000 P b92fc40cacb846a1990f43fa527acdcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92fc40cacb846a1990f43fa527acdcb" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33183 } }
I20260925 22:40:56.784171 4124 raft_consensus.cc:388] T 00000000000000000000000000000000 P b92fc40cacb846a1990f43fa527acdcb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:40:56.784212 4124 raft_consensus.cc:482] T 00000000000000000000000000000000 P b92fc40cacb846a1990f43fa527acdcb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:40:56.784286 4124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b92fc40cacb846a1990f43fa527acdcb [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:56.785234 4124 raft_consensus.cc:504] T 00000000000000000000000000000000 P b92fc40cacb846a1990f43fa527acdcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92fc40cacb846a1990f43fa527acdcb" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33183 } }
I20260925 22:40:56.785563 4124 leader_election.cc:302] T 00000000000000000000000000000000 P b92fc40cacb846a1990f43fa527acdcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b92fc40cacb846a1990f43fa527acdcb; no voters:
I20260925 22:40:56.785851 4124 leader_election.cc:288] T 00000000000000000000000000000000 P b92fc40cacb846a1990f43fa527acdcb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:40:56.786013 4129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b92fc40cacb846a1990f43fa527acdcb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:56.786273 4129 raft_consensus.cc:686] T 00000000000000000000000000000000 P b92fc40cacb846a1990f43fa527acdcb [term 1 LEADER]: Becoming Leader. State: Replica: b92fc40cacb846a1990f43fa527acdcb, State: Running, Role: LEADER
I20260925 22:40:56.786630 4129 consensus_queue.cc:237] T 00000000000000000000000000000000 P b92fc40cacb846a1990f43fa527acdcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92fc40cacb846a1990f43fa527acdcb" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33183 } }
I20260925 22:40:56.787226 4124 sys_catalog.cc:564] T 00000000000000000000000000000000 P b92fc40cacb846a1990f43fa527acdcb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:56.788496 4131 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92fc40cacb846a1990f43fa527acdcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b92fc40cacb846a1990f43fa527acdcb. Latest consensus state: current_term: 1 leader_uuid: "b92fc40cacb846a1990f43fa527acdcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92fc40cacb846a1990f43fa527acdcb" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33183 } } }
I20260925 22:40:56.788655 4131 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92fc40cacb846a1990f43fa527acdcb [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:56.789043 4138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:56.789422 4130 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92fc40cacb846a1990f43fa527acdcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b92fc40cacb846a1990f43fa527acdcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92fc40cacb846a1990f43fa527acdcb" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33183 } } }
I20260925 22:40:56.789543 4130 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92fc40cacb846a1990f43fa527acdcb [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:56.792315 4138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:40:56.798799 4138 catalog_manager.cc:1347] Generated new cluster ID: 398da8c5725b410fa8d243d065a5a834
I20260925 22:40:56.799060 4138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:56.807186 4138 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:56.808295 4138 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:56.814066 4138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b92fc40cacb846a1990f43fa527acdcb: Generated new TSK 0
I20260925 22:40:56.814883 4138 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
I20260925 22:40:56.846843 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:0
--local_ip_for_outbound_sockets=127.2.74.193
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:33183
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:45851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:56.968914 4150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:56.969283 4150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:56.969367 4150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:56.973713 4150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:56.973840 4150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:56.973917 4150 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 22:40:56.973963 4150 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:40:56.974030 4150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:40:56.980211 4150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:45851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.74.254:33183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:56.981958 4150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:56.983408 4150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:56.996621 4155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:56.996780 4150 server_base.cc:1034] running on GCE node
W20260925 22:40:56.996637 4158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:56.997162 4156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:56.997877 4150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:56.998881 4150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:57.000151 4150 hybrid_clock.cc:648] HybridClock initialized: now 1790376057000112 us; error 57 us; skew 500 ppm
Sep 25 22:40:57 dist-test-slave-csnl krb5kdc[4033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376056, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:57.003566 4150 init.cc:377] Logged in from keytab as kudu/127.2.74.193@KRBTEST.COM (short username kudu)
I20260925 22:40:57.004705 4150 webserver.cc:460] Webserver started at http://127.2.74.193:41677/ using document root <none> and password file <none>
I20260925 22:40:57.005388 4150 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:57.005499 4150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:57.005769 4150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:57.008448 4150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-0/data/instance:
uuid: "5786a53a9309463fbcda8fa320957ea9"
format_stamp: "Formatted at 2026-09-25 22:40:57 on dist-test-slave-csnl"
server_key: "fb29424ca2222ff4f09c189afc3f0b64"
server_key_iv: "96d4be35ba2d3a0505e2e3df9dafe87d"
server_key_version: "encryptionkey@0"
I20260925 22:40:57.009279 4150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance:
uuid: "5786a53a9309463fbcda8fa320957ea9"
format_stamp: "Formatted at 2026-09-25 22:40:57 on dist-test-slave-csnl"
server_key: "fb29424ca2222ff4f09c189afc3f0b64"
server_key_iv: "96d4be35ba2d3a0505e2e3df9dafe87d"
server_key_version: "encryptionkey@0"
I20260925 22:40:57.014319 4150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 22:40:57.017755 4165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:57.019623 4150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:40:57.020071 4150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "5786a53a9309463fbcda8fa320957ea9"
format_stamp: "Formatted at 2026-09-25 22:40:57 on dist-test-slave-csnl"
server_key: "fb29424ca2222ff4f09c189afc3f0b64"
server_key_iv: "96d4be35ba2d3a0505e2e3df9dafe87d"
server_key_version: "encryptionkey@0"
I20260925 22:40:57.020253 4150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:57.033744 4150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:57.037030 4150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:57.037274 4150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:57.038764 4150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:57.038848 4150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:57.038925 4150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:57.039011 4150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 25 22:40:57 dist-test-slave-csnl krb5kdc[4033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376056, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:40:57.050271 4150 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:34679
I20260925 22:40:57.050319 4280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:34679 every 8 connection(s)
I20260925 22:40:57.051301 4150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:40:57.054225 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 4150
I20260925 22:40:57.054355 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance
I20260925 22:40:57.054679 2347 external_mini_cluster.cc:1428] Setting key d1036866880805dedab632b0d615214e
I20260925 22:40:57.054702 4127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.039845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:42123 (local address 127.2.74.254:33183)
0925 22:40:57.040015 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:57.040020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:57.041076 (+ 1056us) server_negotiation.cc:407] Connection header received
0925 22:40:57.042576 (+ 1500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:57.042582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:57.042658 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:57.042759 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:57.044767 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.045453 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.047086 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.047357 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.047488 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.050669 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:57.050673 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:57.050676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:57.050715 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:57.052476 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:57.053094 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:57.053097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:57.053098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:57.053179 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:57.053486 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:57.053488 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:57.053489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:57.053669 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:57.053762 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:57.054423 (+ 661us) server_negotiation.cc:299] Negotiation successful
0925 22:40:57.054563 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 22:40:57.055812 4174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.040314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33183 (local address 127.2.74.193:42123)
0925 22:40:57.040839 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:57.040902 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:57.042220 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:57.043089 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:57.043102 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:57.043696 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:57.044543 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:57.044570 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.045649 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.045653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:57.046865 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:57.046878 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.047539 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.047544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:57.047740 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.048443 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:57.048470 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:57.050464 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:57.052606 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:57.052614 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:57.052631 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:57.052976 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:57.053262 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:57.053264 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:57.053266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:57.053404 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:57.053771 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:57.053776 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:57.054099 (+ 323us) client_negotiation.cc:769] Sending connection context
0925 22:40:57.054344 (+ 245us) client_negotiation.cc:240] Negotiation successful
0925 22:40:57.054732 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":125,"threads_started":1}
I20260925 22:40:57.057169 4281 heartbeater.cc:347] Connected to a master server at 127.2.74.254:33183
I20260925 22:40:57.057358 4281 heartbeater.cc:464] Registering TS with master...
I20260925 22:40:57.057941 4281 heartbeater.cc:511] Master 127.2.74.254:33183 requested a full tablet report, sending...
I20260925 22:40:57.060031 4077 ts_manager.cc:194] Registered new tserver with Master: 5786a53a9309463fbcda8fa320957ea9 (127.2.74.193:34679)
I20260925 22:40:57.061450 4077 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.193@KRBTEST.COM'} at 127.2.74.193:42123
WARNING: no policy specified for kudu/127.2.74.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.194@KRBTEST.COM" created.
I20260925 22:40:57.068508 4288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.059818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33183 (local address 127.2.74.193:43741)
0925 22:40:57.060216 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:57.060234 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:57.060382 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:57.060664 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:57.060670 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:57.060945 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:57.061138 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:57.061145 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.062130 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.062137 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:57.063268 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:57.063281 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.063899 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.063903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:57.064012 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.064564 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:57.064582 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:57.065099 (+ 517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:57.066862 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:57.066865 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:57.066867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:57.067162 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:57.067432 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:57.067434 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:57.067436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:57.067511 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:57.067919 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:57.067925 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:57.068074 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 22:40:57.068198 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 22:40:57.068340 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":102,"threads_started":1}
I20260925 22:40:57.068666 4127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.059887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:43741 (local address 127.2.74.254:33183)
0925 22:40:57.060008 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:57.060013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:57.060303 (+ 290us) server_negotiation.cc:407] Connection header received
0925 22:40:57.060482 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:57.060486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:57.060528 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:57.060616 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:57.061276 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.061975 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.063417 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.063742 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.063859 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.065262 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:57.065265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:57.065267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:57.065290 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:57.066760 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:57.067273 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:57.067276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:57.067277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:57.067351 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:57.067571 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:57.067573 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:57.067574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:57.067823 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:57.067928 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:57.068289 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 22:40:57.068432 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
I20260925 22:40:57.083868 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:0
--local_ip_for_outbound_sockets=127.2.74.194
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:33183
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:45851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:57.200016 4290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:57.200310 4290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:57.200361 4290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:57.204638 4290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:57.204720 4290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:57.204763 4290 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 22:40:57.204778 4290 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:40:57.204871 4290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:40:57.210737 4290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:45851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.74.254:33183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:57.211920 4290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:57.213017 4290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:57.223310 4295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:57.223323 4296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:57.223392 4290 server_base.cc:1034] running on GCE node
W20260925 22:40:57.223310 4298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:57.224069 4290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:57.224823 4290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:57.226038 4290 hybrid_clock.cc:648] HybridClock initialized: now 1790376057226014 us; error 44 us; skew 500 ppm
Sep 25 22:40:57 dist-test-slave-csnl krb5kdc[4033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376057, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:57.229277 4290 init.cc:377] Logged in from keytab as kudu/127.2.74.194@KRBTEST.COM (short username kudu)
I20260925 22:40:57.230508 4290 webserver.cc:460] Webserver started at http://127.2.74.194:33495/ using document root <none> and password file <none>
I20260925 22:40:57.231210 4290 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:57.231317 4290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:57.231572 4290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:57.233860 4290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-1/data/instance:
uuid: "42cc1f16cdf64ad78464d04c1cf796a7"
format_stamp: "Formatted at 2026-09-25 22:40:57 on dist-test-slave-csnl"
server_key: "009f3797da32d99e06d10101e148a08e"
server_key_iv: "9010e28c16a981e6cd6c7cc074970b7d"
server_key_version: "encryptionkey@0"
I20260925 22:40:57.234519 4290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance:
uuid: "42cc1f16cdf64ad78464d04c1cf796a7"
format_stamp: "Formatted at 2026-09-25 22:40:57 on dist-test-slave-csnl"
server_key: "009f3797da32d99e06d10101e148a08e"
server_key_iv: "9010e28c16a981e6cd6c7cc074970b7d"
server_key_version: "encryptionkey@0"
I20260925 22:40:57.238875 4290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 22:40:57.241885 4305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:57.243454 4290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:40:57.243619 4290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "42cc1f16cdf64ad78464d04c1cf796a7"
format_stamp: "Formatted at 2026-09-25 22:40:57 on dist-test-slave-csnl"
server_key: "009f3797da32d99e06d10101e148a08e"
server_key_iv: "9010e28c16a981e6cd6c7cc074970b7d"
server_key_version: "encryptionkey@0"
I20260925 22:40:57.243762 4290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:57.257447 4290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:57.260927 4290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:57.261173 4290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:57.262769 4290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:57.262847 4290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:57.262928 4290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:57.263001 4290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 25 22:40:57 dist-test-slave-csnl krb5kdc[4033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376057, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:40:57.274488 4290 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:46367
I20260925 22:40:57.274569 4420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:46367 every 8 connection(s)
I20260925 22:40:57.275555 4290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:40:57.277808 4127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.263685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:42255 (local address 127.2.74.254:33183)
0925 22:40:57.263845 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:57.263850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:57.264906 (+ 1056us) server_negotiation.cc:407] Connection header received
0925 22:40:57.266258 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:57.266262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:57.266332 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:57.266438 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:57.268398 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.268939 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.270413 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.270742 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.270863 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.273694 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:57.273699 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:57.273703 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:57.273738 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:57.275526 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:57.276140 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:57.276144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:57.276146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:57.276239 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:57.276608 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:57.276611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:57.276613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:57.276824 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:57.276915 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:57.277483 (+ 568us) server_negotiation.cc:299] Negotiation successful
0925 22:40:57.277633 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 22:40:57.278633 4314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.264169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33183 (local address 127.2.74.194:42255)
0925 22:40:57.264709 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:57.264772 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:57.265976 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:57.266723 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:57.266732 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:57.267322 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:57.268165 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:57.268192 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.269093 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.269096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:57.270257 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:57.270270 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.270928 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.270933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:57.271151 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.271877 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:57.271897 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:57.273459 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:57.275674 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:57.275680 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:57.275697 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:57.275973 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:57.276343 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:57.276347 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:57.276349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:57.276505 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:57.276926 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:57.276931 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:57.277207 (+ 276us) client_negotiation.cc:769] Sending connection context
0925 22:40:57.277432 (+ 225us) client_negotiation.cc:240] Negotiation successful
0925 22:40:57.277773 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":116,"threads_started":1}
I20260925 22:40:57.280015 4421 heartbeater.cc:347] Connected to a master server at 127.2.74.254:33183
I20260925 22:40:57.280180 4421 heartbeater.cc:464] Registering TS with master...
I20260925 22:40:57.280812 4421 heartbeater.cc:511] Master 127.2.74.254:33183 requested a full tablet report, sending...
I20260925 22:40:57.280936 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 4290
I20260925 22:40:57.281032 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance
I20260925 22:40:57.281250 2347 external_mini_cluster.cc:1428] Setting key 2ab51dbdf018f3b42cfb2b2bcb628aa4
I20260925 22:40:57.281987 4077 ts_manager.cc:194] Registered new tserver with Master: 42cc1f16cdf64ad78464d04c1cf796a7 (127.2.74.194:46367)
I20260925 22:40:57.282763 4077 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.194@KRBTEST.COM'} at 127.2.74.194:42255
WARNING: no policy specified for kudu/127.2.74.195@KRBTEST.COM; defaulting to no policy
I20260925 22:40:57.291764 4127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.283067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:53697 (local address 127.2.74.254:33183)
0925 22:40:57.283213 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:57.283219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:57.283236 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:40:57.283279 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:57.283285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:57.283339 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:57.283426 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:57.284166 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.284878 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.286208 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.286417 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.286494 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.288092 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:57.288096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:57.288099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:57.288142 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:57.289946 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:57.290430 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:57.290432 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:57.290433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:57.290506 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:57.290809 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:57.290811 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:57.290812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:57.291096 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:57.291188 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:57.291539 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 22:40:57.291640 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:40:57.291872 4427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.282205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33183 (local address 127.2.74.194:53697)
0925 22:40:57.282506 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:57.282521 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:57.282612 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:57.283577 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:57.283583 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:57.283848 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:57.284007 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:57.284019 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.285041 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.285048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:57.286079 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:57.286094 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.286541 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.286547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:57.286682 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.287297 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:57.287316 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:57.287856 (+ 540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:57.290060 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:57.290064 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:57.290066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:57.290328 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:57.290599 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:57.290603 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:57.290605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:57.290712 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:57.291202 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:57.291206 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:57.291319 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 22:40:57.291490 (+ 171us) client_negotiation.cc:240] Negotiation successful
0925 22:40:57.291670 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":98,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
I20260925 22:40:57.309643 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:0
--local_ip_for_outbound_sockets=127.2.74.195
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:33183
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:45851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:57.437562 4430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:57.437876 4430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:57.437973 4430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:57.442507 4430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:57.442600 4430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:57.442644 4430 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 22:40:57.442659 4430 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:40:57.442750 4430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:40:57.448714 4430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:45851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.74.254:33183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:57.449872 4430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:57.451067 4430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:57.461218 4436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:57.461231 4435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:57.461386 4430 server_base.cc:1034] running on GCE node
W20260925 22:40:57.461292 4438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:57.461982 4430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:57.462720 4430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:57.463922 4430 hybrid_clock.cc:648] HybridClock initialized: now 1790376057463888 us; error 47 us; skew 500 ppm
Sep 25 22:40:57 dist-test-slave-csnl krb5kdc[4033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376057, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:57.466813 4430 init.cc:377] Logged in from keytab as kudu/127.2.74.195@KRBTEST.COM (short username kudu)
I20260925 22:40:57.467787 4430 webserver.cc:460] Webserver started at http://127.2.74.195:37869/ using document root <none> and password file <none>
I20260925 22:40:57.468380 4430 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:57.468473 4430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:57.468734 4430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:57.470897 4430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-2/data/instance:
uuid: "2708f67c85c34e819ac68d960508e759"
format_stamp: "Formatted at 2026-09-25 22:40:57 on dist-test-slave-csnl"
server_key: "fdb447762137b3ed090eeb35d5ff1913"
server_key_iv: "fee6c8a3d074d91ddedc374796653588"
server_key_version: "encryptionkey@0"
I20260925 22:40:57.471591 4430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance:
uuid: "2708f67c85c34e819ac68d960508e759"
format_stamp: "Formatted at 2026-09-25 22:40:57 on dist-test-slave-csnl"
server_key: "fdb447762137b3ed090eeb35d5ff1913"
server_key_iv: "fee6c8a3d074d91ddedc374796653588"
server_key_version: "encryptionkey@0"
I20260925 22:40:57.475611 4430 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:40:57.478398 4445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:57.480095 4430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:40:57.480261 4430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "2708f67c85c34e819ac68d960508e759"
format_stamp: "Formatted at 2026-09-25 22:40:57 on dist-test-slave-csnl"
server_key: "fdb447762137b3ed090eeb35d5ff1913"
server_key_iv: "fee6c8a3d074d91ddedc374796653588"
server_key_version: "encryptionkey@0"
I20260925 22:40:57.480413 4430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:57.500373 4430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:57.504163 4430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:57.504379 4430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:57.505882 4430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:57.505977 4430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:57.506053 4430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:57.506103 4430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:40:57 dist-test-slave-csnl krb5kdc[4033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376057, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:40:57.517599 4430 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:43423
I20260925 22:40:57.517683 4560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:43423 every 8 connection(s)
I20260925 22:40:57.518625 4430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:40:57.522274 4127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.506880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:34505 (local address 127.2.74.254:33183)
0925 22:40:57.507042 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:57.507046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:57.508309 (+ 1263us) server_negotiation.cc:407] Connection header received
0925 22:40:57.509665 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:57.509668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:57.509735 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:57.509875 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:57.511861 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.512489 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.514427 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.514657 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.514750 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.518185 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:57.518191 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:57.518197 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:57.518236 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:57.519988 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:57.520659 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:57.520662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:57.520663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:57.520750 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:57.521151 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:57.521154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:57.521156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:57.521366 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:57.521452 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:57.522015 (+ 563us) server_negotiation.cc:299] Negotiation successful
0925 22:40:57.522147 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:40:57.523286 4454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.507578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33183 (local address 127.2.74.195:34505)
0925 22:40:57.508133 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:57.508183 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:57.509294 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:57.510150 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:57.510161 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:57.510737 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:57.511651 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:57.511677 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.512612 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.512617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:57.514280 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:57.514292 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.514840 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.514846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:57.515138 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.515941 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:57.515958 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:57.517968 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:57.520110 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:57.520115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:57.520129 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:57.520533 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:57.520835 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:57.520837 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:57.520838 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:57.520999 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:57.521483 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:57.521489 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:57.521758 (+ 269us) client_negotiation.cc:769] Sending connection context
0925 22:40:57.521924 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 22:40:57.522240 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":156,"threads_started":1}
I20260925 22:40:57.524605 4561 heartbeater.cc:347] Connected to a master server at 127.2.74.254:33183
I20260925 22:40:57.524775 4561 heartbeater.cc:464] Registering TS with master...
I20260925 22:40:57.525415 4561 heartbeater.cc:511] Master 127.2.74.254:33183 requested a full tablet report, sending...
I20260925 22:40:57.526719 4077 ts_manager.cc:194] Registered new tserver with Master: 2708f67c85c34e819ac68d960508e759 (127.2.74.195:43423)
I20260925 22:40:57.526728 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 4430
I20260925 22:40:57.526849 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance
I20260925 22:40:57.527249 2347 external_mini_cluster.cc:1428] Setting key d79e6d5c0b1d99c72324c11fffd53339
I20260925 22:40:57.527599 4077 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.195@KRBTEST.COM'} at 127.2.74.195:34505
I20260925 22:40:57.529424 2347 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:40:57.535830 4567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.527264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33183 (local address 127.2.74.195:59449)
0925 22:40:57.527610 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:57.527627 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:57.527731 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:57.528204 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:57.528206 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:57.528450 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:57.528565 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:57.528573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.529534 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.529541 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:57.530191 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:57.530201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.530738 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.530742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:57.530839 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.531491 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:57.531512 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:57.532048 (+ 536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:57.534041 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:57.534049 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:57.534053 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:57.534378 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:57.534677 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:57.534680 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:57.534684 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:57.534788 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:57.535358 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:57.535360 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:57.535453 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 22:40:57.535542 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 22:40:57.535676 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":106,"threads_started":1}
I20260925 22:40:57.536036 4127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.527659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:59449 (local address 127.2.74.254:33183)
0925 22:40:57.527973 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:57.527978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:57.527994 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:40:57.528032 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:57.528036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:57.528080 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:57.528164 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:57.528673 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.529380 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.530321 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.530620 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.530731 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.532247 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:57.532250 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:57.532252 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:57.532277 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:57.533869 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:57.534485 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:57.534487 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:57.534489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:57.534571 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:57.534901 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:57.534906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:57.534908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:57.535248 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:57.535364 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:57.535640 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 22:40:57.535786 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
Sep 25 22:40:57 dist-test-slave-csnl krb5kdc[4033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:40:57 dist-test-slave-csnl krb5kdc[4033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:40:57.557721 4127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.549267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52304 (local address 127.2.74.254:33183)
0925 22:40:57.549406 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:57.549410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:57.549422 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:40:57.549465 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:57.549467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:57.549505 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:57.549601 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:57.550374 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.550901 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.551817 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.552042 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.552129 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.554727 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:57.554731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:57.554733 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:57.554764 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:57.556118 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:57.556592 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:57.556595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:57.556597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:57.556685 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:57.556907 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:57.556909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:57.556910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:57.557093 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:57.557199 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:57.557381 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 22:40:57.557488 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:40:57.561043 4078 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:40:57.564206 4078 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:40:57.580329 4578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.573766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:34679 (local address 127.0.0.1:38346)
0925 22:40:57.574694 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:57.574753 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:57.574879 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:57.575562 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:57.575568 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:57.575611 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:57.575783 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:57.575790 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.577512 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.577514 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:57.578699 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:57.578710 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.579724 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.579728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:57.579843 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.579891 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 22:40:57.580003 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 22:40:57.580142 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":57,"spinlock_wait_cycles":2944,"thread_start_us":130,"threads_started":1}
I20260925 22:40:57.580428 4582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.574755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38346 (local address 127.2.74.193:34679)
0925 22:40:57.575063 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:57.575070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:57.575095 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:40:57.575171 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:57.575177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:57.575416 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:57.575599 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:57.575916 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.577171 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.578830 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.579593 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.579701 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.579860 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:57.580148 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 22:40:57.580260 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":110,"threads_started":1}
I20260925 22:40:57.580539 4579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.574136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:43423 (local address 127.0.0.1:49146)
0925 22:40:57.575510 (+ 1374us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:57.575526 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:57.575621 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:57.576110 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:57.576113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:57.576130 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:57.576222 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:57.576229 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.577218 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.577222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:57.578699 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:57.578710 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.579988 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.579991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:57.580118 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.580138 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 22:40:57.580214 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 22:40:57.580283 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1296,"mutex_wait_us":51,"spinlock_wait_cycles":1408,"thread_start_us":94,"threads_started":1}
I20260925 22:40:57.580879 4581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.573881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:46367 (local address 127.0.0.1:42180)
0925 22:40:57.575174 (+ 1293us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:57.575190 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:57.575280 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:57.575773 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:57.575776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:57.575791 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:57.575876 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:57.575882 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.577293 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.577296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:57.578522 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:57.578531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.579937 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.579941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:57.580034 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.580604 (+ 570us) client_negotiation.cc:769] Sending connection context
0925 22:40:57.580677 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 22:40:57.580747 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"mutex_wait_us":657,"thread_start_us":97,"threads_started":1}
I20260925 22:40:57.581295 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.575143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49146 (local address 127.2.74.195:43423)
0925 22:40:57.575445 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:57.575456 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:57.575710 (+ 254us) server_negotiation.cc:407] Connection header received
0925 22:40:57.575787 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:57.575792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:57.575965 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:57.576119 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:57.576344 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.577093 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.579079 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.579889 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.580004 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.580161 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:57.580244 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 22:40:57.580338 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":100,"threads_started":1}
I20260925 22:40:57.581341 4580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.574064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.2.74.194:46367)
0925 22:40:57.574662 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:57.574674 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:57.575381 (+ 707us) server_negotiation.cc:407] Connection header received
0925 22:40:57.575464 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:57.575473 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:57.575653 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:57.575804 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:57.576011 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.577171 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.578659 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.579591 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.579702 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.579873 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:57.581082 (+ 1209us) server_negotiation.cc:299] Negotiation successful
0925 22:40:57.581189 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":113,"threads_started":1}
I20260925 22:40:57.583657 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 94229a61aec34a19be9a5e4cc87b603e (DEFAULT_TABLE table=test-table [id=f97f09e4c3144ed7ac3b53785ccc9fe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:57.584024 4355 tablet_service.cc:1467] Processing CreateTablet for tablet 94229a61aec34a19be9a5e4cc87b603e (DEFAULT_TABLE table=test-table [id=f97f09e4c3144ed7ac3b53785ccc9fe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:57.584820 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94229a61aec34a19be9a5e4cc87b603e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:57.585088 4355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94229a61aec34a19be9a5e4cc87b603e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:57.584765 4495 tablet_service.cc:1467] Processing CreateTablet for tablet 94229a61aec34a19be9a5e4cc87b603e (DEFAULT_TABLE table=test-table [id=f97f09e4c3144ed7ac3b53785ccc9fe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:57.585713 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94229a61aec34a19be9a5e4cc87b603e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:57.591638 4584 tablet_bootstrap.cc:492] T 94229a61aec34a19be9a5e4cc87b603e P 5786a53a9309463fbcda8fa320957ea9: Bootstrap starting.
I20260925 22:40:57.593241 4585 tablet_bootstrap.cc:492] T 94229a61aec34a19be9a5e4cc87b603e P 42cc1f16cdf64ad78464d04c1cf796a7: Bootstrap starting.
I20260925 22:40:57.594316 4586 tablet_bootstrap.cc:492] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759: Bootstrap starting.
I20260925 22:40:57.595289 4584 tablet_bootstrap.cc:654] T 94229a61aec34a19be9a5e4cc87b603e P 5786a53a9309463fbcda8fa320957ea9: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:57.595289 4585 tablet_bootstrap.cc:654] T 94229a61aec34a19be9a5e4cc87b603e P 42cc1f16cdf64ad78464d04c1cf796a7: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:57.596134 4586 tablet_bootstrap.cc:654] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:57.596225 4584 log.cc:824] T 94229a61aec34a19be9a5e4cc87b603e P 5786a53a9309463fbcda8fa320957ea9: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:57.596225 4585 log.cc:824] T 94229a61aec34a19be9a5e4cc87b603e P 42cc1f16cdf64ad78464d04c1cf796a7: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:57.596979 4586 log.cc:824] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:57.598752 4585 tablet_bootstrap.cc:492] T 94229a61aec34a19be9a5e4cc87b603e P 42cc1f16cdf64ad78464d04c1cf796a7: No bootstrap required, opened a new log
I20260925 22:40:57.598986 4586 tablet_bootstrap.cc:492] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759: No bootstrap required, opened a new log
I20260925 22:40:57.599051 4585 ts_tablet_manager.cc:1397] T 94229a61aec34a19be9a5e4cc87b603e P 42cc1f16cdf64ad78464d04c1cf796a7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:40:57.599176 4586 ts_tablet_manager.cc:1397] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260925 22:40:57.599484 4584 tablet_bootstrap.cc:492] T 94229a61aec34a19be9a5e4cc87b603e P 5786a53a9309463fbcda8fa320957ea9: No bootstrap required, opened a new log
I20260925 22:40:57.599669 4584 ts_tablet_manager.cc:1397] T 94229a61aec34a19be9a5e4cc87b603e P 5786a53a9309463fbcda8fa320957ea9: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 22:40:57.602247 4585 raft_consensus.cc:348] T 94229a61aec34a19be9a5e4cc87b603e P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:57.602533 4585 raft_consensus.cc:374] T 94229a61aec34a19be9a5e4cc87b603e P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:57.602589 4585 raft_consensus.cc:729] T 94229a61aec34a19be9a5e4cc87b603e P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cc1f16cdf64ad78464d04c1cf796a7, State: Initialized, Role: FOLLOWER
I20260925 22:40:57.603150 4585 consensus_queue.cc:260] T 94229a61aec34a19be9a5e4cc87b603e P 42cc1f16cdf64ad78464d04c1cf796a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:57.603404 4586 raft_consensus.cc:348] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:57.603703 4586 raft_consensus.cc:374] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:57.603765 4586 raft_consensus.cc:729] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2708f67c85c34e819ac68d960508e759, State: Initialized, Role: FOLLOWER
I20260925 22:40:57.604044 4421 heartbeater.cc:503] Master 127.2.74.254:33183 was elected leader, sending a full tablet report...
I20260925 22:40:57.604249 4586 consensus_queue.cc:260] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:57.604321 4585 ts_tablet_manager.cc:1428] T 94229a61aec34a19be9a5e4cc87b603e P 42cc1f16cdf64ad78464d04c1cf796a7: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:40:57.604223 4584 raft_consensus.cc:348] T 94229a61aec34a19be9a5e4cc87b603e P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:57.604472 4584 raft_consensus.cc:374] T 94229a61aec34a19be9a5e4cc87b603e P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:57.604547 4584 raft_consensus.cc:729] T 94229a61aec34a19be9a5e4cc87b603e P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5786a53a9309463fbcda8fa320957ea9, State: Initialized, Role: FOLLOWER
I20260925 22:40:57.604996 4584 consensus_queue.cc:260] T 94229a61aec34a19be9a5e4cc87b603e P 5786a53a9309463fbcda8fa320957ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:57.605324 4586 ts_tablet_manager.cc:1428] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:40:57.605680 4561 heartbeater.cc:503] Master 127.2.74.254:33183 was elected leader, sending a full tablet report...
I20260925 22:40:57.605986 4584 ts_tablet_manager.cc:1428] T 94229a61aec34a19be9a5e4cc87b603e P 5786a53a9309463fbcda8fa320957ea9: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260925 22:40:57.606302 4281 heartbeater.cc:503] Master 127.2.74.254:33183 was elected leader, sending a full tablet report...
I20260925 22:40:57.732661 4591 raft_consensus.cc:482] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:40:57.732896 4591 raft_consensus.cc:504] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:57.734160 4591 leader_election.cc:288] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5786a53a9309463fbcda8fa320957ea9 (127.2.74.193:34679), 42cc1f16cdf64ad78464d04c1cf796a7 (127.2.74.194:46367)
I20260925 22:40:57.737876 4454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.734452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:46367 (local address 127.2.74.195:47197)
0925 22:40:57.734644 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:57.734660 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:57.734766 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:57.735077 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:57.735080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:57.735097 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:57.735217 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:57.735222 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.735968 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.735973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:57.736768 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:57.736778 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.737433 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.737436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:57.737543 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.737564 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:40:57.737658 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 22:40:57.737739 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260925 22:40:57.737876 4580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.734552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:47197 (local address 127.2.74.194:46367)
0925 22:40:57.734660 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:57.734665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:57.734710 (+ 45us) server_negotiation.cc:407] Connection header received
0925 22:40:57.734887 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:57.734890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:57.734992 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:57.735064 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:57.735321 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.735850 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.736893 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.737311 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.737441 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.737513 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:57.737682 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 22:40:57.737747 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:40:57.738550 4375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94229a61aec34a19be9a5e4cc87b603e" candidate_uuid: "2708f67c85c34e819ac68d960508e759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" is_pre_election: true
I20260925 22:40:57.738735 4593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.734495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:34679 (local address 127.2.74.195:44289)
0925 22:40:57.734998 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:57.735014 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:57.735156 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:57.735411 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:57.735414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:57.735429 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:57.735566 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:57.735571 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.736272 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.736278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:57.737434 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:57.737443 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.738300 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.738304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:57.738404 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.738422 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 22:40:57.738518 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 22:40:57.738594 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":149,"threads_started":1}
I20260925 22:40:57.738722 4582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.734647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:44289 (local address 127.2.74.193:34679)
0925 22:40:57.734778 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:57.734783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:57.735111 (+ 328us) server_negotiation.cc:407] Connection header received
0925 22:40:57.735269 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:57.735272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:57.735316 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:57.735402 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:57.735680 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.736164 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.737574 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.738073 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.738187 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.738232 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:57.738535 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 22:40:57.738610 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:40:57.738909 4375 raft_consensus.cc:2454] T 94229a61aec34a19be9a5e4cc87b603e P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2708f67c85c34e819ac68d960508e759 in term 0.
I20260925 22:40:57.739274 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94229a61aec34a19be9a5e4cc87b603e" candidate_uuid: "2708f67c85c34e819ac68d960508e759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5786a53a9309463fbcda8fa320957ea9" is_pre_election: true
I20260925 22:40:57.739511 4235 raft_consensus.cc:2454] T 94229a61aec34a19be9a5e4cc87b603e P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2708f67c85c34e819ac68d960508e759 in term 0.
I20260925 22:40:57.739477 4448 leader_election.cc:302] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2708f67c85c34e819ac68d960508e759, 42cc1f16cdf64ad78464d04c1cf796a7; no voters:
I20260925 22:40:57.739779 4591 raft_consensus.cc:2781] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:40:57.739892 4591 raft_consensus.cc:482] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:40:57.739954 4591 raft_consensus.cc:3037] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:57.741020 4591 raft_consensus.cc:504] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:57.741438 4591 leader_election.cc:288] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759 [CANDIDATE]: Term 1 election: Requested vote from peers 5786a53a9309463fbcda8fa320957ea9 (127.2.74.193:34679), 42cc1f16cdf64ad78464d04c1cf796a7 (127.2.74.194:46367)
I20260925 22:40:57.741830 4375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94229a61aec34a19be9a5e4cc87b603e" candidate_uuid: "2708f67c85c34e819ac68d960508e759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cc1f16cdf64ad78464d04c1cf796a7"
I20260925 22:40:57.741827 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94229a61aec34a19be9a5e4cc87b603e" candidate_uuid: "2708f67c85c34e819ac68d960508e759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5786a53a9309463fbcda8fa320957ea9"
I20260925 22:40:57.741966 4375 raft_consensus.cc:3037] T 94229a61aec34a19be9a5e4cc87b603e P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:57.741962 4235 raft_consensus.cc:3037] T 94229a61aec34a19be9a5e4cc87b603e P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:57.743322 4235 raft_consensus.cc:2454] T 94229a61aec34a19be9a5e4cc87b603e P 5786a53a9309463fbcda8fa320957ea9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2708f67c85c34e819ac68d960508e759 in term 1.
I20260925 22:40:57.743322 4375 raft_consensus.cc:2454] T 94229a61aec34a19be9a5e4cc87b603e P 42cc1f16cdf64ad78464d04c1cf796a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2708f67c85c34e819ac68d960508e759 in term 1.
I20260925 22:40:57.743717 4446 leader_election.cc:302] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2708f67c85c34e819ac68d960508e759, 42cc1f16cdf64ad78464d04c1cf796a7, 5786a53a9309463fbcda8fa320957ea9; no voters:
I20260925 22:40:57.743973 4591 raft_consensus.cc:2781] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:57.744191 4591 raft_consensus.cc:686] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759 [term 1 LEADER]: Becoming Leader. State: Replica: 2708f67c85c34e819ac68d960508e759, State: Running, Role: LEADER
I20260925 22:40:57.744549 4591 consensus_queue.cc:237] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:57.747964 4076 catalog_manager.cc:5495] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2708f67c85c34e819ac68d960508e759 (127.2.74.195). New cstate: current_term: 1 leader_uuid: "2708f67c85c34e819ac68d960508e759" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } health_report { overall_health: HEALTHY } } }
W20260925 22:40:57.769959 4562 tablet.cc:2369] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:40:57.776852 4422 tablet.cc:2369] T 94229a61aec34a19be9a5e4cc87b603e P 42cc1f16cdf64ad78464d04c1cf796a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 22:40:57 dist-test-slave-csnl krb5kdc[4033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376056, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:40:57.797415 4601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.788805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33183 (local address 127.0.0.1:52312)
0925 22:40:57.789209 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:57.789221 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:57.789372 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:57.789622 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:57.789624 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:57.790017 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:57.790128 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:57.790133 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.790828 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.790830 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:57.791468 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:57.791477 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.791924 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.791926 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:57.791991 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.792598 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:57.792609 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:57.794125 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:57.795888 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:57.795896 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:57.795901 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:57.796151 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:57.796419 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:57.796421 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:57.796431 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:57.796508 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:57.796883 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:57.796889 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:57.796994 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 22:40:57.797101 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 22:40:57.797229 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":200,"threads_started":1}
I20260925 22:40:57.797420 4127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.788924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52312 (local address 127.2.74.254:33183)
0925 22:40:57.789100 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:57.789106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:57.789308 (+ 202us) server_negotiation.cc:407] Connection header received
0925 22:40:57.789459 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:57.789462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:57.789502 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:57.789570 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:57.790229 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.790736 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.791596 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.791817 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.791905 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.794256 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:57.794259 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:57.794261 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:57.794281 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:57.795779 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:57.796254 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:57.796257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:57.796258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:57.796336 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:57.796585 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:57.796587 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:57.796588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:57.796766 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:57.796882 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:57.797160 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 22:40:57.797289 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":48}
I20260925 22:40:57.801631 4076 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:52312:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 22:40:57.802490 4076 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260925 22:40:57.802753 4282 tablet.cc:2369] T 94229a61aec34a19be9a5e4cc87b603e P 5786a53a9309463fbcda8fa320957ea9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:40:57.809878 4495 tablet_service.cc:1467] Processing CreateTablet for tablet c5c2089cdeb54ee1b6916e68ccd92f8f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f87c6af99e67495795335fbb2c898c71]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:40:57.809621 4215 tablet_service.cc:1467] Processing CreateTablet for tablet c5c2089cdeb54ee1b6916e68ccd92f8f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f87c6af99e67495795335fbb2c898c71]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:40:57.809828 4355 tablet_service.cc:1467] Processing CreateTablet for tablet c5c2089cdeb54ee1b6916e68ccd92f8f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f87c6af99e67495795335fbb2c898c71]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:40:57.810576 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5c2089cdeb54ee1b6916e68ccd92f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:57.810619 4355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5c2089cdeb54ee1b6916e68ccd92f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:57.810629 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5c2089cdeb54ee1b6916e68ccd92f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:57.813742 4586 tablet_bootstrap.cc:492] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759: Bootstrap starting.
I20260925 22:40:57.814282 4584 tablet_bootstrap.cc:492] T c5c2089cdeb54ee1b6916e68ccd92f8f P 5786a53a9309463fbcda8fa320957ea9: Bootstrap starting.
I20260925 22:40:57.815155 4586 tablet_bootstrap.cc:654] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:57.815518 4584 tablet_bootstrap.cc:654] T c5c2089cdeb54ee1b6916e68ccd92f8f P 5786a53a9309463fbcda8fa320957ea9: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:57.816112 4585 tablet_bootstrap.cc:492] T c5c2089cdeb54ee1b6916e68ccd92f8f P 42cc1f16cdf64ad78464d04c1cf796a7: Bootstrap starting.
I20260925 22:40:57.817122 4585 tablet_bootstrap.cc:654] T c5c2089cdeb54ee1b6916e68ccd92f8f P 42cc1f16cdf64ad78464d04c1cf796a7: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:57.818722 4585 tablet_bootstrap.cc:492] T c5c2089cdeb54ee1b6916e68ccd92f8f P 42cc1f16cdf64ad78464d04c1cf796a7: No bootstrap required, opened a new log
I20260925 22:40:57.818815 4585 ts_tablet_manager.cc:1397] T c5c2089cdeb54ee1b6916e68ccd92f8f P 42cc1f16cdf64ad78464d04c1cf796a7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:40:57.818858 4586 tablet_bootstrap.cc:492] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759: No bootstrap required, opened a new log
I20260925 22:40:57.818938 4586 ts_tablet_manager.cc:1397] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 22:40:57.819327 4585 raft_consensus.cc:348] T c5c2089cdeb54ee1b6916e68ccd92f8f P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:57.819453 4585 raft_consensus.cc:374] T c5c2089cdeb54ee1b6916e68ccd92f8f P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:57.819443 4584 tablet_bootstrap.cc:492] T c5c2089cdeb54ee1b6916e68ccd92f8f P 5786a53a9309463fbcda8fa320957ea9: No bootstrap required, opened a new log
I20260925 22:40:57.819399 4586 raft_consensus.cc:348] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:57.819487 4585 raft_consensus.cc:729] T c5c2089cdeb54ee1b6916e68ccd92f8f P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cc1f16cdf64ad78464d04c1cf796a7, State: Initialized, Role: FOLLOWER
I20260925 22:40:57.819497 4586 raft_consensus.cc:374] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:57.819531 4586 raft_consensus.cc:729] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2708f67c85c34e819ac68d960508e759, State: Initialized, Role: FOLLOWER
I20260925 22:40:57.819543 4584 ts_tablet_manager.cc:1397] T c5c2089cdeb54ee1b6916e68ccd92f8f P 5786a53a9309463fbcda8fa320957ea9: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260925 22:40:57.819645 4585 consensus_queue.cc:260] T c5c2089cdeb54ee1b6916e68ccd92f8f P 42cc1f16cdf64ad78464d04c1cf796a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:57.819645 4586 consensus_queue.cc:260] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:57.819912 4585 ts_tablet_manager.cc:1428] T c5c2089cdeb54ee1b6916e68ccd92f8f P 42cc1f16cdf64ad78464d04c1cf796a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260925 22:40:57.819912 4586 ts_tablet_manager.cc:1428] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 22:40:57.820099 4584 raft_consensus.cc:348] T c5c2089cdeb54ee1b6916e68ccd92f8f P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:57.820241 4584 raft_consensus.cc:374] T c5c2089cdeb54ee1b6916e68ccd92f8f P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:57.820178 4591 tablet_replica.cc:439] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } } }
I20260925 22:40:57.820281 4584 raft_consensus.cc:729] T c5c2089cdeb54ee1b6916e68ccd92f8f P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5786a53a9309463fbcda8fa320957ea9, State: Initialized, Role: FOLLOWER
I20260925 22:40:57.820320 4591 tablet_replica.cc:442] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:40:57.820480 4584 consensus_queue.cc:260] T c5c2089cdeb54ee1b6916e68ccd92f8f P 5786a53a9309463fbcda8fa320957ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:57.820555 4590 tablet_replica.cc:439] T c5c2089cdeb54ee1b6916e68ccd92f8f P 42cc1f16cdf64ad78464d04c1cf796a7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } } }
I20260925 22:40:57.820717 4590 tablet_replica.cc:442] T c5c2089cdeb54ee1b6916e68ccd92f8f P 42cc1f16cdf64ad78464d04c1cf796a7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:40:57.820751 4592 tablet_replica.cc:439] T c5c2089cdeb54ee1b6916e68ccd92f8f P 5786a53a9309463fbcda8fa320957ea9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } } }
I20260925 22:40:57.820885 4592 tablet_replica.cc:442] T c5c2089cdeb54ee1b6916e68ccd92f8f P 5786a53a9309463fbcda8fa320957ea9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:40:57.822391 4584 ts_tablet_manager.cc:1428] T c5c2089cdeb54ee1b6916e68ccd92f8f P 5786a53a9309463fbcda8fa320957ea9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 22:40:57.932911 4591 raft_consensus.cc:482] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:40:57.933072 4591 raft_consensus.cc:504] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:57.933548 4591 leader_election.cc:288] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5786a53a9309463fbcda8fa320957ea9 (127.2.74.193:34679), 42cc1f16cdf64ad78464d04c1cf796a7 (127.2.74.194:46367)
I20260925 22:40:57.934113 4375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2089cdeb54ee1b6916e68ccd92f8f" candidate_uuid: "2708f67c85c34e819ac68d960508e759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" is_pre_election: true
I20260925 22:40:57.934113 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2089cdeb54ee1b6916e68ccd92f8f" candidate_uuid: "2708f67c85c34e819ac68d960508e759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5786a53a9309463fbcda8fa320957ea9" is_pre_election: true
I20260925 22:40:57.934280 4375 raft_consensus.cc:2454] T c5c2089cdeb54ee1b6916e68ccd92f8f P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2708f67c85c34e819ac68d960508e759 in term 0.
I20260925 22:40:57.934279 4235 raft_consensus.cc:2454] T c5c2089cdeb54ee1b6916e68ccd92f8f P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2708f67c85c34e819ac68d960508e759 in term 0.
I20260925 22:40:57.934585 4448 leader_election.cc:302] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2708f67c85c34e819ac68d960508e759, 42cc1f16cdf64ad78464d04c1cf796a7; no voters:
I20260925 22:40:57.934787 4591 raft_consensus.cc:2781] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:40:57.934850 4591 raft_consensus.cc:482] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:40:57.934877 4591 raft_consensus.cc:3037] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:57.935927 4591 raft_consensus.cc:504] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:57.936321 4591 leader_election.cc:288] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759 [CANDIDATE]: Term 1 election: Requested vote from peers 5786a53a9309463fbcda8fa320957ea9 (127.2.74.193:34679), 42cc1f16cdf64ad78464d04c1cf796a7 (127.2.74.194:46367)
I20260925 22:40:57.936734 4375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2089cdeb54ee1b6916e68ccd92f8f" candidate_uuid: "2708f67c85c34e819ac68d960508e759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cc1f16cdf64ad78464d04c1cf796a7"
I20260925 22:40:57.936756 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2089cdeb54ee1b6916e68ccd92f8f" candidate_uuid: "2708f67c85c34e819ac68d960508e759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5786a53a9309463fbcda8fa320957ea9"
I20260925 22:40:57.936900 4235 raft_consensus.cc:3037] T c5c2089cdeb54ee1b6916e68ccd92f8f P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:57.936899 4375 raft_consensus.cc:3037] T c5c2089cdeb54ee1b6916e68ccd92f8f P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:57.937836 4235 raft_consensus.cc:2454] T c5c2089cdeb54ee1b6916e68ccd92f8f P 5786a53a9309463fbcda8fa320957ea9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2708f67c85c34e819ac68d960508e759 in term 1.
I20260925 22:40:57.937855 4375 raft_consensus.cc:2454] T c5c2089cdeb54ee1b6916e68ccd92f8f P 42cc1f16cdf64ad78464d04c1cf796a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2708f67c85c34e819ac68d960508e759 in term 1.
I20260925 22:40:57.938128 4446 leader_election.cc:302] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2708f67c85c34e819ac68d960508e759, 5786a53a9309463fbcda8fa320957ea9; no voters:
I20260925 22:40:57.938324 4591 raft_consensus.cc:2781] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:57.938418 4591 raft_consensus.cc:686] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759 [term 1 LEADER]: Becoming Leader. State: Replica: 2708f67c85c34e819ac68d960508e759, State: Running, Role: LEADER
I20260925 22:40:57.938565 4591 consensus_queue.cc:237] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:57.939306 4594 tablet_replica.cc:439] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759: TxnStatusTablet state changed. Reason: New leader 2708f67c85c34e819ac68d960508e759. Latest consensus state: current_term: 1 leader_uuid: "2708f67c85c34e819ac68d960508e759" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } } }
I20260925 22:40:57.939437 4594 tablet_replica.cc:442] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:40:57.939747 4606 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 22:40:57.940001 4606 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 22:40:57.940454 4076 catalog_manager.cc:5495] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2708f67c85c34e819ac68d960508e759 (127.2.74.195). New cstate: current_term: 1 leader_uuid: "2708f67c85c34e819ac68d960508e759" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } health_report { overall_health: HEALTHY } } }
I20260925 22:40:57.954674 4601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.950492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:43423 (local address 127.0.0.1:49150)
0925 22:40:57.950656 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:57.950673 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:57.950774 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:57.951164 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:57.951167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:57.951176 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:40:57.951270 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:57.951275 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.952267 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.952270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:57.952891 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:57.952899 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.953466 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.953468 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:57.953533 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.953580 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:40:57.954385 (+ 805us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:40:57.954399 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 22:40:57.954453 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 22:40:57.954540 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 22:40:57.954697 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.950600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49150 (local address 127.2.74.195:43423)
0925 22:40:57.950732 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:57.950736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:57.950753 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:40:57.950913 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:57.950917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:57.951032 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:57.951146 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:40:57.951368 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.952163 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.953002 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.953353 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.953460 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.953719 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:40:57.954285 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:40:57.954401 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:57.954468 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 22:40:57.954535 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:40:57.960551 4235 raft_consensus.cc:1261] T c5c2089cdeb54ee1b6916e68ccd92f8f P 5786a53a9309463fbcda8fa320957ea9 [term 1 FOLLOWER]: Refusing update from remote peer 2708f67c85c34e819ac68d960508e759: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:40:57.960561 4375 raft_consensus.cc:1261] T c5c2089cdeb54ee1b6916e68ccd92f8f P 42cc1f16cdf64ad78464d04c1cf796a7 [term 1 FOLLOWER]: Refusing update from remote peer 2708f67c85c34e819ac68d960508e759: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:40:57.961400 4591 consensus_queue.cc:1035] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:40:57.961648 4594 consensus_queue.cc:1035] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:40:57.966205 4592 tablet_replica.cc:439] T c5c2089cdeb54ee1b6916e68ccd92f8f P 5786a53a9309463fbcda8fa320957ea9: TxnStatusTablet state changed. Reason: New leader 2708f67c85c34e819ac68d960508e759. Latest consensus state: current_term: 1 leader_uuid: "2708f67c85c34e819ac68d960508e759" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } } }
I20260925 22:40:57.966429 4592 tablet_replica.cc:442] T c5c2089cdeb54ee1b6916e68ccd92f8f P 5786a53a9309463fbcda8fa320957ea9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:40:57.967133 4590 tablet_replica.cc:439] T c5c2089cdeb54ee1b6916e68ccd92f8f P 42cc1f16cdf64ad78464d04c1cf796a7: TxnStatusTablet state changed. Reason: New leader 2708f67c85c34e819ac68d960508e759. Latest consensus state: current_term: 1 leader_uuid: "2708f67c85c34e819ac68d960508e759" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } } }
I20260925 22:40:57.967273 4590 tablet_replica.cc:442] T c5c2089cdeb54ee1b6916e68ccd92f8f P 42cc1f16cdf64ad78464d04c1cf796a7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:40:57.970085 4591 tablet_replica.cc:439] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2708f67c85c34e819ac68d960508e759" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } } }
I20260925 22:40:57.970237 4591 tablet_replica.cc:442] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:40:57.970362 4609 mvcc.cc:205] Tried to move back new op lower bound from 7333380333398671360 to 7333380333317525504. Current Snapshot: MvccSnapshot[applied={T|T < 7333380333398671360}]
I20260925 22:40:57.970897 4594 tablet_replica.cc:439] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2708f67c85c34e819ac68d960508e759" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } } }
I20260925 22:40:57.971033 4594 tablet_replica.cc:442] T c5c2089cdeb54ee1b6916e68ccd92f8f P 2708f67c85c34e819ac68d960508e759: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:40:57.971818 4590 tablet_replica.cc:439] T c5c2089cdeb54ee1b6916e68ccd92f8f P 42cc1f16cdf64ad78464d04c1cf796a7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2708f67c85c34e819ac68d960508e759" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } } }
I20260925 22:40:57.971939 4590 tablet_replica.cc:442] T c5c2089cdeb54ee1b6916e68ccd92f8f P 42cc1f16cdf64ad78464d04c1cf796a7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:40:57.973323 4592 tablet_replica.cc:439] T c5c2089cdeb54ee1b6916e68ccd92f8f P 5786a53a9309463fbcda8fa320957ea9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2708f67c85c34e819ac68d960508e759" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } } }
I20260925 22:40:57.973464 4592 tablet_replica.cc:442] T c5c2089cdeb54ee1b6916e68ccd92f8f P 5786a53a9309463fbcda8fa320957ea9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:40:57.975296 4610 mvcc.cc:205] Tried to move back new op lower bound from 7333380333398671360 to 7333380333317525504. Current Snapshot: MvccSnapshot[applied={T|T < 7333380333398671360}]
I20260925 22:40:57.975844 4611 mvcc.cc:205] Tried to move back new op lower bound from 7333380333398671360 to 7333380333317525504. Current Snapshot: MvccSnapshot[applied={T|T < 7333380333398671360}]
I20260925 22:40:57.987628 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.984090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49158 (local address 127.2.74.195:43423)
0925 22:40:57.984260 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:57.984264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:57.984297 (+ 33us) server_negotiation.cc:407] Connection header received
0925 22:40:57.984501 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:57.984504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:57.984551 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:57.984669 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:40:57.984952 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.985526 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.986431 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.986662 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.986773 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.987151 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:40:57.987235 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:40:57.987323 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:57.987418 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 22:40:57.987472 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 22:40:58.000047 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.996275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:48931 (local address 127.2.74.195:43423)
0925 22:40:57.996401 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:57.996405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:57.996475 (+ 70us) server_negotiation.cc:407] Connection header received
0925 22:40:57.996605 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:57.996608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:57.996662 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:57.996751 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:40:57.996974 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.997755 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.998679 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.999063 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.999185 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.999520 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:40:57.999611 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:40:57.999684 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:57.999800 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 22:40:57.999879 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":56}
I20260925 22:40:58.000028 4567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:57.996215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:43423 (local address 127.2.74.195:48931)
0925 22:40:57.996376 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:57.996394 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:57.996511 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:57.996765 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:57.996767 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:57.996772 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:40:57.996875 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:57.996879 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.997896 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.997899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:57.998549 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:57.998558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:57.999219 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:57.999222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:57.999301 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:57.999355 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:40:57.999701 (+ 346us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:40:57.999714 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 22:40:57.999765 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 22:40:57.999883 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":26}
I20260925 22:40:58.008442 4235 raft_consensus.cc:1261] T 94229a61aec34a19be9a5e4cc87b603e P 5786a53a9309463fbcda8fa320957ea9 [term 1 FOLLOWER]: Refusing update from remote peer 2708f67c85c34e819ac68d960508e759: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:40:58.008478 4375 raft_consensus.cc:1261] T 94229a61aec34a19be9a5e4cc87b603e P 42cc1f16cdf64ad78464d04c1cf796a7 [term 1 FOLLOWER]: Refusing update from remote peer 2708f67c85c34e819ac68d960508e759: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:40:58.008889 4620 consensus_queue.cc:1035] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:40:58.009060 4594 consensus_queue.cc:1035] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:40:58.062683 4078 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:40:58.062898 4078 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:40:58.065804 4078 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b92fc40cacb846a1990f43fa527acdcb: Sending DeleteTablet for 3 replicas of tablet 94229a61aec34a19be9a5e4cc87b603e
I20260925 22:40:58.066795 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet 94229a61aec34a19be9a5e4cc87b603e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:58 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:38346
I20260925 22:40:58.067128 4355 tablet_service.cc:1514] Processing DeleteTablet for tablet 94229a61aec34a19be9a5e4cc87b603e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:58 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:42180
I20260925 22:40:58.067696 4625 tablet_replica.cc:330] T 94229a61aec34a19be9a5e4cc87b603e P 42cc1f16cdf64ad78464d04c1cf796a7: stopping tablet replica
I20260925 22:40:58.067834 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet 94229a61aec34a19be9a5e4cc87b603e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:58 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:49146
I20260925 22:40:58.068222 4625 raft_consensus.cc:2229] T 94229a61aec34a19be9a5e4cc87b603e P 42cc1f16cdf64ad78464d04c1cf796a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:40:58.068356 4626 tablet_replica.cc:330] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759: stopping tablet replica
I20260925 22:40:58.068601 4625 raft_consensus.cc:2258] T 94229a61aec34a19be9a5e4cc87b603e P 42cc1f16cdf64ad78464d04c1cf796a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:40:58.068699 4626 raft_consensus.cc:2229] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:40:58.069417 4626 raft_consensus.cc:2258] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:40:58.069758 4627 tablet_replica.cc:330] T 94229a61aec34a19be9a5e4cc87b603e P 5786a53a9309463fbcda8fa320957ea9: stopping tablet replica
I20260925 22:40:58.069725 4078 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 22:40:58.070061 4627 raft_consensus.cc:2229] T 94229a61aec34a19be9a5e4cc87b603e P 5786a53a9309463fbcda8fa320957ea9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:40:58.070266 4627 raft_consensus.cc:2258] T 94229a61aec34a19be9a5e4cc87b603e P 5786a53a9309463fbcda8fa320957ea9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 22:40:58.070254 4078 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:40:58.070595 4625 ts_tablet_manager.cc:1905] T 94229a61aec34a19be9a5e4cc87b603e P 42cc1f16cdf64ad78464d04c1cf796a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:40:58.071903 4626 ts_tablet_manager.cc:1905] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:40:58.073215 4627 ts_tablet_manager.cc:1905] T 94229a61aec34a19be9a5e4cc87b603e P 5786a53a9309463fbcda8fa320957ea9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:40:58.075295 4626 ts_tablet_manager.cc:1918] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:40:58.075421 4626 log.cc:1196] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-2/wal/wals/94229a61aec34a19be9a5e4cc87b603e
I20260925 22:40:58.075812 4626 ts_tablet_manager.cc:1939] T 94229a61aec34a19be9a5e4cc87b603e P 2708f67c85c34e819ac68d960508e759: Deleting consensus metadata
I20260925 22:40:58.076519 4627 ts_tablet_manager.cc:1918] T 94229a61aec34a19be9a5e4cc87b603e P 5786a53a9309463fbcda8fa320957ea9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:40:58.076637 4627 log.cc:1196] T 94229a61aec34a19be9a5e4cc87b603e P 5786a53a9309463fbcda8fa320957ea9: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-0/wal/wals/94229a61aec34a19be9a5e4cc87b603e
I20260925 22:40:58.076942 4627 ts_tablet_manager.cc:1939] T 94229a61aec34a19be9a5e4cc87b603e P 5786a53a9309463fbcda8fa320957ea9: Deleting consensus metadata
I20260925 22:40:58.077423 4215 tablet_service.cc:1467] Processing CreateTablet for tablet f3ac3c04fc1f4655b064d61c1b61cda9 (DEFAULT_TABLE table=test-table [id=7ac2bb9029aa4e819422cb2acfa18313]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:58.077579 4355 tablet_service.cc:1467] Processing CreateTablet for tablet f3ac3c04fc1f4655b064d61c1b61cda9 (DEFAULT_TABLE table=test-table [id=7ac2bb9029aa4e819422cb2acfa18313]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:58.077662 4065 catalog_manager.cc:4841] TS 2708f67c85c34e819ac68d960508e759 (127.2.74.195:43423): tablet 94229a61aec34a19be9a5e4cc87b603e (table test-table [id=f97f09e4c3144ed7ac3b53785ccc9fe5]) successfully deleted
I20260925 22:40:58.077703 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3ac3c04fc1f4655b064d61c1b61cda9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:58.077883 4355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3ac3c04fc1f4655b064d61c1b61cda9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:58.080132 4625 ts_tablet_manager.cc:1918] T 94229a61aec34a19be9a5e4cc87b603e P 42cc1f16cdf64ad78464d04c1cf796a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:40:58.080258 4585 tablet_bootstrap.cc:492] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7: Bootstrap starting.
I20260925 22:40:58.080356 4625 log.cc:1196] T 94229a61aec34a19be9a5e4cc87b603e P 42cc1f16cdf64ad78464d04c1cf796a7: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-1/wal/wals/94229a61aec34a19be9a5e4cc87b603e
I20260925 22:40:58.080565 4494 tablet_service.cc:1467] Processing CreateTablet for tablet f3ac3c04fc1f4655b064d61c1b61cda9 (DEFAULT_TABLE table=test-table [id=7ac2bb9029aa4e819422cb2acfa18313]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:58.080818 4625 ts_tablet_manager.cc:1939] T 94229a61aec34a19be9a5e4cc87b603e P 42cc1f16cdf64ad78464d04c1cf796a7: Deleting consensus metadata
I20260925 22:40:58.080873 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3ac3c04fc1f4655b064d61c1b61cda9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:58.081354 4585 tablet_bootstrap.cc:654] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:58.082190 4064 catalog_manager.cc:4841] TS 42cc1f16cdf64ad78464d04c1cf796a7 (127.2.74.194:46367): tablet 94229a61aec34a19be9a5e4cc87b603e (table test-table [id=f97f09e4c3144ed7ac3b53785ccc9fe5]) successfully deleted
I20260925 22:40:58.082849 4585 tablet_bootstrap.cc:492] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7: No bootstrap required, opened a new log
I20260925 22:40:58.082968 4585 ts_tablet_manager.cc:1397] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 22:40:58.083290 4584 tablet_bootstrap.cc:492] T f3ac3c04fc1f4655b064d61c1b61cda9 P 5786a53a9309463fbcda8fa320957ea9: Bootstrap starting.
I20260925 22:40:58.083470 4585 raft_consensus.cc:348] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:58.083590 4585 raft_consensus.cc:374] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:58.083622 4585 raft_consensus.cc:729] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cc1f16cdf64ad78464d04c1cf796a7, State: Initialized, Role: FOLLOWER
I20260925 22:40:58.083760 4585 consensus_queue.cc:260] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:58.084045 4585 ts_tablet_manager.cc:1428] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:40:58.084358 4584 tablet_bootstrap.cc:654] T f3ac3c04fc1f4655b064d61c1b61cda9 P 5786a53a9309463fbcda8fa320957ea9: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:58.086282 4584 tablet_bootstrap.cc:492] T f3ac3c04fc1f4655b064d61c1b61cda9 P 5786a53a9309463fbcda8fa320957ea9: No bootstrap required, opened a new log
I20260925 22:40:58.086386 4584 ts_tablet_manager.cc:1397] T f3ac3c04fc1f4655b064d61c1b61cda9 P 5786a53a9309463fbcda8fa320957ea9: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 22:40:58.086877 4584 raft_consensus.cc:348] T f3ac3c04fc1f4655b064d61c1b61cda9 P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:58.087031 4584 raft_consensus.cc:374] T f3ac3c04fc1f4655b064d61c1b61cda9 P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:58.087065 4584 raft_consensus.cc:729] T f3ac3c04fc1f4655b064d61c1b61cda9 P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5786a53a9309463fbcda8fa320957ea9, State: Initialized, Role: FOLLOWER
I20260925 22:40:58.088240 4586 tablet_bootstrap.cc:492] T f3ac3c04fc1f4655b064d61c1b61cda9 P 2708f67c85c34e819ac68d960508e759: Bootstrap starting.
I20260925 22:40:58.088652 4584 consensus_queue.cc:260] T f3ac3c04fc1f4655b064d61c1b61cda9 P 5786a53a9309463fbcda8fa320957ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:58.089798 4586 tablet_bootstrap.cc:654] T f3ac3c04fc1f4655b064d61c1b61cda9 P 2708f67c85c34e819ac68d960508e759: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:58.090366 4584 ts_tablet_manager.cc:1428] T f3ac3c04fc1f4655b064d61c1b61cda9 P 5786a53a9309463fbcda8fa320957ea9: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260925 22:40:58.091284 4586 tablet_bootstrap.cc:492] T f3ac3c04fc1f4655b064d61c1b61cda9 P 2708f67c85c34e819ac68d960508e759: No bootstrap required, opened a new log
I20260925 22:40:58.091403 4586 ts_tablet_manager.cc:1397] T f3ac3c04fc1f4655b064d61c1b61cda9 P 2708f67c85c34e819ac68d960508e759: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20260925 22:40:58.091918 4586 raft_consensus.cc:348] T f3ac3c04fc1f4655b064d61c1b61cda9 P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:58.092051 4586 raft_consensus.cc:374] T f3ac3c04fc1f4655b064d61c1b61cda9 P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:58.092113 4586 raft_consensus.cc:729] T f3ac3c04fc1f4655b064d61c1b61cda9 P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2708f67c85c34e819ac68d960508e759, State: Initialized, Role: FOLLOWER
I20260925 22:40:58.092238 4586 consensus_queue.cc:260] T f3ac3c04fc1f4655b064d61c1b61cda9 P 2708f67c85c34e819ac68d960508e759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:58.092491 4586 ts_tablet_manager.cc:1428] T f3ac3c04fc1f4655b064d61c1b61cda9 P 2708f67c85c34e819ac68d960508e759: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:40:58.093196 4062 catalog_manager.cc:4841] TS 5786a53a9309463fbcda8fa320957ea9 (127.2.74.193:34679): tablet 94229a61aec34a19be9a5e4cc87b603e (table test-table [id=f97f09e4c3144ed7ac3b53785ccc9fe5]) successfully deleted
I20260925 22:40:58.206380 4590 raft_consensus.cc:482] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:40:58.206562 4590 raft_consensus.cc:504] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:58.207789 4590 leader_election.cc:288] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5786a53a9309463fbcda8fa320957ea9 (127.2.74.193:34679), 2708f67c85c34e819ac68d960508e759 (127.2.74.195:43423)
I20260925 22:40:58.208747 4592 raft_consensus.cc:482] T f3ac3c04fc1f4655b064d61c1b61cda9 P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:40:58.208864 4592 raft_consensus.cc:504] T f3ac3c04fc1f4655b064d61c1b61cda9 P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:58.210009 4592 leader_election.cc:288] T f3ac3c04fc1f4655b064d61c1b61cda9 P 5786a53a9309463fbcda8fa320957ea9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cc1f16cdf64ad78464d04c1cf796a7 (127.2.74.194:46367), 2708f67c85c34e819ac68d960508e759 (127.2.74.195:43423)
I20260925 22:40:58.213265 4631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:58.208014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:34679 (local address 127.2.74.194:36503)
0925 22:40:58.208369 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:58.208386 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:58.208553 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:58.208952 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:58.208954 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:58.208968 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:58.209083 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:58.209089 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:58.210165 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:58.210171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:58.211734 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:58.211747 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:58.212795 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:58.212799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:58.212910 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:58.212933 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:40:58.213013 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 22:40:58.213088 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":103,"threads_started":1}
I20260925 22:40:58.213272 4582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:58.208265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:36503 (local address 127.2.74.193:34679)
0925 22:40:58.208398 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:58.208404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:58.208468 (+ 64us) server_negotiation.cc:407] Connection header received
0925 22:40:58.208697 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:58.208702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:58.208755 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:58.208809 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:58.209217 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:58.210006 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:58.211885 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:58.212541 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:58.212624 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:58.212674 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:58.213050 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 22:40:58.213144 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":50}
I20260925 22:40:58.213771 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ac3c04fc1f4655b064d61c1b61cda9" candidate_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5786a53a9309463fbcda8fa320957ea9" is_pre_election: true
I20260925 22:40:58.213943 4235 raft_consensus.cc:2454] T f3ac3c04fc1f4655b064d61c1b61cda9 P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cc1f16cdf64ad78464d04c1cf796a7 in term 0.
I20260925 22:40:58.213984 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:58.208419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:33999 (local address 127.2.74.195:43423)
0925 22:40:58.208538 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:58.208542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:58.208665 (+ 123us) server_negotiation.cc:407] Connection header received
0925 22:40:58.208838 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:58.208841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:58.208889 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:58.208940 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:58.209440 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:58.210141 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:58.211892 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:58.212502 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:58.212630 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:58.212674 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:58.213782 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0925 22:40:58.213861 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":50}
I20260925 22:40:58.214558 4632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:58.208068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:43423 (local address 127.2.74.194:33999)
0925 22:40:58.208571 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:58.208584 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:58.208723 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:58.209189 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:58.209195 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:58.209214 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:58.209320 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:58.209325 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:58.210321 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:58.210324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:58.211769 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:58.211777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:58.212795 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:58.212799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:58.213623 (+ 824us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:58.213652 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 22:40:58.214330 (+ 678us) client_negotiation.cc:240] Negotiation successful
0925 22:40:58.214410 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":688,"thread_start_us":121,"threads_started":1}
I20260925 22:40:58.214747 4306 leader_election.cc:302] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cc1f16cdf64ad78464d04c1cf796a7, 5786a53a9309463fbcda8fa320957ea9; no voters:
I20260925 22:40:58.215070 4590 raft_consensus.cc:2781] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:40:58.215140 4590 raft_consensus.cc:482] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:40:58.215183 4590 raft_consensus.cc:3037] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:58.215518 4515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ac3c04fc1f4655b064d61c1b61cda9" candidate_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2708f67c85c34e819ac68d960508e759" is_pre_election: true
I20260925 22:40:58.215754 4515 raft_consensus.cc:2454] T f3ac3c04fc1f4655b064d61c1b61cda9 P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cc1f16cdf64ad78464d04c1cf796a7 in term 0.
I20260925 22:40:58.216462 4590 raft_consensus.cc:504] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:58.216861 4590 leader_election.cc:288] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7 [CANDIDATE]: Term 1 election: Requested vote from peers 5786a53a9309463fbcda8fa320957ea9 (127.2.74.193:34679), 2708f67c85c34e819ac68d960508e759 (127.2.74.195:43423)
I20260925 22:40:58.217432 4515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ac3c04fc1f4655b064d61c1b61cda9" candidate_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2708f67c85c34e819ac68d960508e759"
I20260925 22:40:58.217443 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ac3c04fc1f4655b064d61c1b61cda9" candidate_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5786a53a9309463fbcda8fa320957ea9"
I20260925 22:40:58.217561 4235 raft_consensus.cc:3037] T f3ac3c04fc1f4655b064d61c1b61cda9 P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:58.217559 4515 raft_consensus.cc:3037] T f3ac3c04fc1f4655b064d61c1b61cda9 P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:58.217634 4580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:58.211247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:46369 (local address 127.2.74.194:46367)
0925 22:40:58.211356 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:58.211361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:58.211498 (+ 137us) server_negotiation.cc:407] Connection header received
0925 22:40:58.211667 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:58.211670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:58.211710 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:58.211751 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:58.214179 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:58.214802 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:58.216286 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:58.216868 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:58.217003 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:58.217062 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:58.217420 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 22:40:58.217503 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 22:40:58.218039 4634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:58.210352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:46367 (local address 127.2.74.193:46369)
0925 22:40:58.211423 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:58.211436 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:58.211559 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:58.211864 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:58.211867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:58.211884 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:58.213787 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:58.213798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:58.214942 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:58.214969 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:58.216154 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:58.216164 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:58.217170 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:58.217175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:58.217274 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:58.217303 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 22:40:58.217801 (+ 498us) client_negotiation.cc:240] Negotiation successful
0925 22:40:58.217891 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1009,"mutex_wait_us":1772,"thread_start_us":77,"threads_started":1}
I20260925 22:40:58.218703 4235 raft_consensus.cc:2454] T f3ac3c04fc1f4655b064d61c1b61cda9 P 5786a53a9309463fbcda8fa320957ea9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cc1f16cdf64ad78464d04c1cf796a7 in term 1.
I20260925 22:40:58.218868 4515 raft_consensus.cc:2454] T f3ac3c04fc1f4655b064d61c1b61cda9 P 2708f67c85c34e819ac68d960508e759 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cc1f16cdf64ad78464d04c1cf796a7 in term 1.
I20260925 22:40:58.219020 4306 leader_election.cc:302] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cc1f16cdf64ad78464d04c1cf796a7, 5786a53a9309463fbcda8fa320957ea9; no voters:
I20260925 22:40:58.219168 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ac3c04fc1f4655b064d61c1b61cda9" candidate_uuid: "5786a53a9309463fbcda8fa320957ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" is_pre_election: true
I20260925 22:40:58.219210 4590 raft_consensus.cc:2781] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:58.219460 4590 raft_consensus.cc:686] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7 [term 1 LEADER]: Becoming Leader. State: Replica: 42cc1f16cdf64ad78464d04c1cf796a7, State: Running, Role: LEADER
I20260925 22:40:58.219725 4590 consensus_queue.cc:237] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:58.221748 4633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:58.210713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:43423 (local address 127.2.74.193:34265)
0925 22:40:58.212083 (+ 1370us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:58.212095 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:58.212177 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:58.212610 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:58.212613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:58.212630 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:58.212735 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:58.212741 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:58.219226 (+ 6485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:58.219230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:58.220510 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:58.220521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:58.221282 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:58.221286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:58.221387 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:58.221420 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 22:40:58.221485 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 22:40:58.221576 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1314,"thread_start_us":101,"threads_started":1}
I20260925 22:40:58.222995 4076 catalog_manager.cc:5495] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42cc1f16cdf64ad78464d04c1cf796a7 (127.2.74.194). New cstate: current_term: 1 leader_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } health_report { overall_health: UNKNOWN } } }
I20260925 22:40:58.223161 4635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:58.211204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:34265 (local address 127.2.74.195:43423)
0925 22:40:58.212354 (+ 1150us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:58.212366 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:58.212377 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:40:58.212414 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:58.212417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:58.212455 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:58.212524 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:58.212861 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:58.218838 (+ 5977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:58.220645 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:58.221156 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:58.222161 (+ 1005us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:58.222228 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:58.222270 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 22:40:58.222355 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":244,"server-negotiator.queue_time_us":1061,"thread_start_us":76,"threads_started":1}
I20260925 22:40:58.223935 4515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ac3c04fc1f4655b064d61c1b61cda9" candidate_uuid: "5786a53a9309463fbcda8fa320957ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2708f67c85c34e819ac68d960508e759" is_pre_election: true
I20260925 22:40:58.224185 4515 raft_consensus.cc:2379] T f3ac3c04fc1f4655b064d61c1b61cda9 P 2708f67c85c34e819ac68d960508e759 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5786a53a9309463fbcda8fa320957ea9 in current term 1: Already voted for candidate 42cc1f16cdf64ad78464d04c1cf796a7 in this term.
I20260925 22:40:58.225134 4169 leader_election.cc:302] T f3ac3c04fc1f4655b064d61c1b61cda9 P 5786a53a9309463fbcda8fa320957ea9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5786a53a9309463fbcda8fa320957ea9; no voters: 2708f67c85c34e819ac68d960508e759, 42cc1f16cdf64ad78464d04c1cf796a7
I20260925 22:40:58.225690 4592 raft_consensus.cc:2726] T f3ac3c04fc1f4655b064d61c1b61cda9 P 5786a53a9309463fbcda8fa320957ea9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 22:40:58.260994 4580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:58.256902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42188 (local address 127.2.74.194:46367)
0925 22:40:58.257047 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:58.257053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:58.257100 (+ 47us) server_negotiation.cc:407] Connection header received
0925 22:40:58.257303 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:58.257306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:58.257371 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:58.257512 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:40:58.257754 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:58.258406 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:58.259478 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:58.259709 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:58.259822 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:58.260107 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:40:58.260582 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:40:58.260714 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:58.260774 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 22:40:58.260830 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 22:40:58.273159 4640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:58.268939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:43423 (local address 127.2.74.194:48925)
0925 22:40:58.269268 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:58.269281 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:58.269377 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:58.269613 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:58.269615 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:58.269621 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:40:58.269739 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:58.269743 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:58.270812 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:58.270815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:58.271659 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:58.271671 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:58.272288 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:58.272291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:58.272393 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:58.272444 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:40:58.272848 (+ 404us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:40:58.272855 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 22:40:58.272919 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 22:40:58.272995 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":138,"threads_started":1}
I20260925 22:40:58.273161 4635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:58.269009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:48925 (local address 127.2.74.195:43423)
0925 22:40:58.269156 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:58.269161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:58.269349 (+ 188us) server_negotiation.cc:407] Connection header received
0925 22:40:58.269453 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:58.269456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:58.269510 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:58.269594 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:40:58.269837 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:58.270651 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:58.271791 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:58.272153 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:58.272261 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:58.272606 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:40:58.272719 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:40:58.272816 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:58.272955 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 22:40:58.273015 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:40:58.281829 4515 raft_consensus.cc:1261] T f3ac3c04fc1f4655b064d61c1b61cda9 P 2708f67c85c34e819ac68d960508e759 [term 1 FOLLOWER]: Refusing update from remote peer 42cc1f16cdf64ad78464d04c1cf796a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:40:58.282321 4235 raft_consensus.cc:1261] T f3ac3c04fc1f4655b064d61c1b61cda9 P 5786a53a9309463fbcda8fa320957ea9 [term 1 FOLLOWER]: Refusing update from remote peer 42cc1f16cdf64ad78464d04c1cf796a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:40:58.282766 4590 consensus_queue.cc:1035] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:40:58.283149 4636 consensus_queue.cc:1035] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:40:58.303372 4567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:58.299909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:46367 (local address 127.2.74.195:51565)
0925 22:40:58.300033 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:58.300049 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:58.300145 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:58.300423 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:58.300426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:58.300433 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:40:58.300543 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:58.300548 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:58.301501 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:58.301508 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:58.302112 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:58.302123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:58.302608 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:58.302612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:58.302708 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:58.302735 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:40:58.303071 (+ 336us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:40:58.303081 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 22:40:58.303163 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 22:40:58.303256 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 22:40:58.303521 4580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:58.299973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:51565 (local address 127.2.74.194:46367)
0925 22:40:58.300083 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:58.300090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:58.300105 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:40:58.300266 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:58.300269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:58.300317 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:58.300415 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:40:58.300659 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:58.301393 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:58.302249 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:58.302489 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:58.302615 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:58.302856 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:40:58.302941 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:40:58.303061 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:58.303336 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 22:40:58.303396 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260925 22:40:58.304313 4611 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 22:40:59.878376 4078 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:40:59.878567 4078 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:40:59.881349 4078 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b92fc40cacb846a1990f43fa527acdcb: Sending DeleteTablet for 3 replicas of tablet f3ac3c04fc1f4655b064d61c1b61cda9
I20260925 22:40:59.881963 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet f3ac3c04fc1f4655b064d61c1b61cda9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:59 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:38346
I20260925 22:40:59.882012 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet f3ac3c04fc1f4655b064d61c1b61cda9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:59 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:49146
I20260925 22:40:59.882012 4355 tablet_service.cc:1514] Processing DeleteTablet for tablet f3ac3c04fc1f4655b064d61c1b61cda9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:59 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:42180
I20260925 22:40:59.882313 4666 tablet_replica.cc:330] T f3ac3c04fc1f4655b064d61c1b61cda9 P 5786a53a9309463fbcda8fa320957ea9: stopping tablet replica
I20260925 22:40:59.882426 4668 tablet_replica.cc:330] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7: stopping tablet replica
I20260925 22:40:59.882453 4666 raft_consensus.cc:2229] T f3ac3c04fc1f4655b064d61c1b61cda9 P 5786a53a9309463fbcda8fa320957ea9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:40:59.882581 4666 raft_consensus.cc:2258] T f3ac3c04fc1f4655b064d61c1b61cda9 P 5786a53a9309463fbcda8fa320957ea9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:40:59.882583 4668 raft_consensus.cc:2229] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:40:59.882949 4668 raft_consensus.cc:2258] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:40:59.883177 4667 tablet_replica.cc:330] T f3ac3c04fc1f4655b064d61c1b61cda9 P 2708f67c85c34e819ac68d960508e759: stopping tablet replica
I20260925 22:40:59.883356 4667 raft_consensus.cc:2229] T f3ac3c04fc1f4655b064d61c1b61cda9 P 2708f67c85c34e819ac68d960508e759 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:40:59.883522 4667 raft_consensus.cc:2258] T f3ac3c04fc1f4655b064d61c1b61cda9 P 2708f67c85c34e819ac68d960508e759 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:40:59.883909 4668 ts_tablet_manager.cc:1905] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:40:59.884249 4667 ts_tablet_manager.cc:1905] T f3ac3c04fc1f4655b064d61c1b61cda9 P 2708f67c85c34e819ac68d960508e759: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:40:59.886087 4666 ts_tablet_manager.cc:1905] T f3ac3c04fc1f4655b064d61c1b61cda9 P 5786a53a9309463fbcda8fa320957ea9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:40:59.886829 4668 ts_tablet_manager.cc:1918] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:40:59.886916 4668 log.cc:1196] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-1/wal/wals/f3ac3c04fc1f4655b064d61c1b61cda9
I20260925 22:40:59.887290 4668 ts_tablet_manager.cc:1939] T f3ac3c04fc1f4655b064d61c1b61cda9 P 42cc1f16cdf64ad78464d04c1cf796a7: Deleting consensus metadata
I20260925 22:40:59.887661 4667 ts_tablet_manager.cc:1918] T f3ac3c04fc1f4655b064d61c1b61cda9 P 2708f67c85c34e819ac68d960508e759: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:40:59.887820 4667 log.cc:1196] T f3ac3c04fc1f4655b064d61c1b61cda9 P 2708f67c85c34e819ac68d960508e759: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-2/wal/wals/f3ac3c04fc1f4655b064d61c1b61cda9
I20260925 22:40:59.888139 4064 catalog_manager.cc:4841] TS 42cc1f16cdf64ad78464d04c1cf796a7 (127.2.74.194:46367): tablet f3ac3c04fc1f4655b064d61c1b61cda9 (table test-table [id=7ac2bb9029aa4e819422cb2acfa18313]) successfully deleted
I20260925 22:40:59.888214 4667 ts_tablet_manager.cc:1939] T f3ac3c04fc1f4655b064d61c1b61cda9 P 2708f67c85c34e819ac68d960508e759: Deleting consensus metadata
I20260925 22:40:59.888998 4065 catalog_manager.cc:4841] TS 2708f67c85c34e819ac68d960508e759 (127.2.74.195:43423): tablet f3ac3c04fc1f4655b064d61c1b61cda9 (table test-table [id=7ac2bb9029aa4e819422cb2acfa18313]) successfully deleted
I20260925 22:40:59.889495 4666 ts_tablet_manager.cc:1918] T f3ac3c04fc1f4655b064d61c1b61cda9 P 5786a53a9309463fbcda8fa320957ea9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:40:59.889565 4666 log.cc:1196] T f3ac3c04fc1f4655b064d61c1b61cda9 P 5786a53a9309463fbcda8fa320957ea9: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-0/wal/wals/f3ac3c04fc1f4655b064d61c1b61cda9
I20260925 22:40:59.889823 4666 ts_tablet_manager.cc:1939] T f3ac3c04fc1f4655b064d61c1b61cda9 P 5786a53a9309463fbcda8fa320957ea9: Deleting consensus metadata
I20260925 22:40:59.890493 4062 catalog_manager.cc:4841] TS 5786a53a9309463fbcda8fa320957ea9 (127.2.74.193:34679): tablet f3ac3c04fc1f4655b064d61c1b61cda9 (table test-table [id=7ac2bb9029aa4e819422cb2acfa18313]) successfully deleted
Sep 25 22:40:59 dist-test-slave-csnl krb5kdc[4033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376059, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:40:59 dist-test-slave-csnl krb5kdc[4033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376059, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:40:59.911936 4676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:59.902578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52316 (local address 127.2.74.254:33183)
0925 22:40:59.902895 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:59.902904 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:59.903027 (+ 123us) server_negotiation.cc:407] Connection header received
0925 22:40:59.903232 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:59.903235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:59.903283 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:59.903389 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:59.904122 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:59.904679 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:59.905567 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:59.905912 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:59.906038 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:59.908655 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:59.908658 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:59.908661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:59.908689 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:59.910100 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:59.910572 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:59.910576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:59.910578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:59.910675 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:59.910981 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:59.910987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:59.910989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:59.911197 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:59.911347 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:59.911602 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 22:40:59.911749 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":104,"threads_started":1}
I20260925 22:40:59.914608 4078 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:40:59.915126 4078 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:40:59.920192 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 3c629099418c49a3b3122c66cc6ff0ec (DEFAULT_TABLE table=test-table [id=88f4a0910f814a7fb05330bd96a8656f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:59.920209 4355 tablet_service.cc:1467] Processing CreateTablet for tablet 3c629099418c49a3b3122c66cc6ff0ec (DEFAULT_TABLE table=test-table [id=88f4a0910f814a7fb05330bd96a8656f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:59.920455 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 3c629099418c49a3b3122c66cc6ff0ec (DEFAULT_TABLE table=test-table [id=88f4a0910f814a7fb05330bd96a8656f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:59.920537 4355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c629099418c49a3b3122c66cc6ff0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:59.920542 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c629099418c49a3b3122c66cc6ff0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:59.920722 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c629099418c49a3b3122c66cc6ff0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:59.922848 4677 tablet_bootstrap.cc:492] T 3c629099418c49a3b3122c66cc6ff0ec P 5786a53a9309463fbcda8fa320957ea9: Bootstrap starting.
I20260925 22:40:59.923406 4679 tablet_bootstrap.cc:492] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7: Bootstrap starting.
I20260925 22:40:59.923717 4678 tablet_bootstrap.cc:492] T 3c629099418c49a3b3122c66cc6ff0ec P 2708f67c85c34e819ac68d960508e759: Bootstrap starting.
I20260925 22:40:59.923884 4677 tablet_bootstrap.cc:654] T 3c629099418c49a3b3122c66cc6ff0ec P 5786a53a9309463fbcda8fa320957ea9: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:59.924569 4679 tablet_bootstrap.cc:654] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:59.924599 4678 tablet_bootstrap.cc:654] T 3c629099418c49a3b3122c66cc6ff0ec P 2708f67c85c34e819ac68d960508e759: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:59.925217 4677 tablet_bootstrap.cc:492] T 3c629099418c49a3b3122c66cc6ff0ec P 5786a53a9309463fbcda8fa320957ea9: No bootstrap required, opened a new log
I20260925 22:40:59.925343 4677 ts_tablet_manager.cc:1397] T 3c629099418c49a3b3122c66cc6ff0ec P 5786a53a9309463fbcda8fa320957ea9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:40:59.925743 4678 tablet_bootstrap.cc:492] T 3c629099418c49a3b3122c66cc6ff0ec P 2708f67c85c34e819ac68d960508e759: No bootstrap required, opened a new log
I20260925 22:40:59.925851 4678 ts_tablet_manager.cc:1397] T 3c629099418c49a3b3122c66cc6ff0ec P 2708f67c85c34e819ac68d960508e759: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:40:59.925837 4677 raft_consensus.cc:348] T 3c629099418c49a3b3122c66cc6ff0ec P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:59.925968 4677 raft_consensus.cc:374] T 3c629099418c49a3b3122c66cc6ff0ec P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:59.926012 4677 raft_consensus.cc:729] T 3c629099418c49a3b3122c66cc6ff0ec P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5786a53a9309463fbcda8fa320957ea9, State: Initialized, Role: FOLLOWER
I20260925 22:40:59.926162 4677 consensus_queue.cc:260] T 3c629099418c49a3b3122c66cc6ff0ec P 5786a53a9309463fbcda8fa320957ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:59.926311 4678 raft_consensus.cc:348] T 3c629099418c49a3b3122c66cc6ff0ec P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:59.926433 4678 raft_consensus.cc:374] T 3c629099418c49a3b3122c66cc6ff0ec P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:59.926481 4678 raft_consensus.cc:729] T 3c629099418c49a3b3122c66cc6ff0ec P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2708f67c85c34e819ac68d960508e759, State: Initialized, Role: FOLLOWER
I20260925 22:40:59.926571 4677 ts_tablet_manager.cc:1428] T 3c629099418c49a3b3122c66cc6ff0ec P 5786a53a9309463fbcda8fa320957ea9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:40:59.926630 4678 consensus_queue.cc:260] T 3c629099418c49a3b3122c66cc6ff0ec P 2708f67c85c34e819ac68d960508e759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:59.926909 4678 ts_tablet_manager.cc:1428] T 3c629099418c49a3b3122c66cc6ff0ec P 2708f67c85c34e819ac68d960508e759: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:40:59.927518 4679 tablet_bootstrap.cc:492] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7: No bootstrap required, opened a new log
I20260925 22:40:59.927636 4679 ts_tablet_manager.cc:1397] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 22:40:59.927997 4679 raft_consensus.cc:348] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:59.928117 4679 raft_consensus.cc:374] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:59.928157 4679 raft_consensus.cc:729] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cc1f16cdf64ad78464d04c1cf796a7, State: Initialized, Role: FOLLOWER
I20260925 22:40:59.928298 4679 consensus_queue.cc:260] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:40:59.928655 4679 ts_tablet_manager.cc:1428] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:41:00.144305 4659 raft_consensus.cc:482] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:41:00.144464 4659 raft_consensus.cc:504] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:41:00.145015 4659 leader_election.cc:288] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5786a53a9309463fbcda8fa320957ea9 (127.2.74.193:34679), 2708f67c85c34e819ac68d960508e759 (127.2.74.195:43423)
I20260925 22:41:00.145574 4515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c629099418c49a3b3122c66cc6ff0ec" candidate_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2708f67c85c34e819ac68d960508e759" is_pre_election: true
I20260925 22:41:00.145574 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c629099418c49a3b3122c66cc6ff0ec" candidate_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5786a53a9309463fbcda8fa320957ea9" is_pre_election: true
I20260925 22:41:00.145749 4515 raft_consensus.cc:2454] T 3c629099418c49a3b3122c66cc6ff0ec P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cc1f16cdf64ad78464d04c1cf796a7 in term 0.
I20260925 22:41:00.145750 4235 raft_consensus.cc:2454] T 3c629099418c49a3b3122c66cc6ff0ec P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cc1f16cdf64ad78464d04c1cf796a7 in term 0.
I20260925 22:41:00.146174 4306 leader_election.cc:302] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2708f67c85c34e819ac68d960508e759, 42cc1f16cdf64ad78464d04c1cf796a7, 5786a53a9309463fbcda8fa320957ea9; no voters:
I20260925 22:41:00.146363 4659 raft_consensus.cc:2781] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:41:00.146422 4659 raft_consensus.cc:482] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:41:00.146472 4659 raft_consensus.cc:3037] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:00.147843 4659 raft_consensus.cc:504] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:41:00.148285 4659 leader_election.cc:288] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7 [CANDIDATE]: Term 1 election: Requested vote from peers 5786a53a9309463fbcda8fa320957ea9 (127.2.74.193:34679), 2708f67c85c34e819ac68d960508e759 (127.2.74.195:43423)
I20260925 22:41:00.148705 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c629099418c49a3b3122c66cc6ff0ec" candidate_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5786a53a9309463fbcda8fa320957ea9"
I20260925 22:41:00.148758 4515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c629099418c49a3b3122c66cc6ff0ec" candidate_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2708f67c85c34e819ac68d960508e759"
I20260925 22:41:00.148835 4235 raft_consensus.cc:3037] T 3c629099418c49a3b3122c66cc6ff0ec P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:00.148926 4515 raft_consensus.cc:3037] T 3c629099418c49a3b3122c66cc6ff0ec P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:00.150329 4235 raft_consensus.cc:2454] T 3c629099418c49a3b3122c66cc6ff0ec P 5786a53a9309463fbcda8fa320957ea9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cc1f16cdf64ad78464d04c1cf796a7 in term 1.
I20260925 22:41:00.150375 4515 raft_consensus.cc:2454] T 3c629099418c49a3b3122c66cc6ff0ec P 2708f67c85c34e819ac68d960508e759 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cc1f16cdf64ad78464d04c1cf796a7 in term 1.
I20260925 22:41:00.150787 4306 leader_election.cc:302] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2708f67c85c34e819ac68d960508e759, 42cc1f16cdf64ad78464d04c1cf796a7, 5786a53a9309463fbcda8fa320957ea9; no voters:
I20260925 22:41:00.151021 4659 raft_consensus.cc:2781] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:41:00.151149 4659 raft_consensus.cc:686] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7 [term 1 LEADER]: Becoming Leader. State: Replica: 42cc1f16cdf64ad78464d04c1cf796a7, State: Running, Role: LEADER
I20260925 22:41:00.151309 4659 consensus_queue.cc:237] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:41:00.153231 4077 catalog_manager.cc:5495] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42cc1f16cdf64ad78464d04c1cf796a7 (127.2.74.194). New cstate: current_term: 1 leader_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } health_report { overall_health: UNKNOWN } } }
I20260925 22:41:00.171382 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:00.167834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42200 (local address 127.2.74.194:46367)
0925 22:41:00.168173 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:00.168180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:00.168191 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:41:00.168231 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:00.168233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:00.168274 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:00.168371 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:41:00.168641 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:00.169292 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:00.170199 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:00.170441 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:00.170565 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:00.170849 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:41:00.170941 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:41:00.171070 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:00.171162 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 22:41:00.171219 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":136,"threads_started":1}
I20260925 22:41:00.178686 4235 raft_consensus.cc:1261] T 3c629099418c49a3b3122c66cc6ff0ec P 5786a53a9309463fbcda8fa320957ea9 [term 1 FOLLOWER]: Refusing update from remote peer 42cc1f16cdf64ad78464d04c1cf796a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:41:00.178686 4515 raft_consensus.cc:1261] T 3c629099418c49a3b3122c66cc6ff0ec P 2708f67c85c34e819ac68d960508e759 [term 1 FOLLOWER]: Refusing update from remote peer 42cc1f16cdf64ad78464d04c1cf796a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:41:00.179134 4659 consensus_queue.cc:1035] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:41:00.179368 4660 consensus_queue.cc:1035] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:41:00.215940 4077 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:41:00.216145 4077 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:41:00.218437 4077 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b92fc40cacb846a1990f43fa527acdcb: Sending DeleteTablet for 3 replicas of tablet 3c629099418c49a3b3122c66cc6ff0ec
I20260925 22:41:00.219213 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c629099418c49a3b3122c66cc6ff0ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:00 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:49146
I20260925 22:41:00.219219 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c629099418c49a3b3122c66cc6ff0ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:00 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:38346
I20260925 22:41:00.219270 4355 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c629099418c49a3b3122c66cc6ff0ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:00 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:42180
I20260925 22:41:00.219452 4668 tablet_replica.cc:330] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7: stopping tablet replica
I20260925 22:41:00.219455 4667 tablet_replica.cc:330] T 3c629099418c49a3b3122c66cc6ff0ec P 2708f67c85c34e819ac68d960508e759: stopping tablet replica
I20260925 22:41:00.219553 4668 raft_consensus.cc:2229] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:41:00.219587 4667 raft_consensus.cc:2229] T 3c629099418c49a3b3122c66cc6ff0ec P 2708f67c85c34e819ac68d960508e759 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:41:00.219745 4667 raft_consensus.cc:2258] T 3c629099418c49a3b3122c66cc6ff0ec P 2708f67c85c34e819ac68d960508e759 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:41:00.219795 4668 raft_consensus.cc:2258] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:41:00.219956 4666 tablet_replica.cc:330] T 3c629099418c49a3b3122c66cc6ff0ec P 5786a53a9309463fbcda8fa320957ea9: stopping tablet replica
I20260925 22:41:00.220078 4666 raft_consensus.cc:2229] T 3c629099418c49a3b3122c66cc6ff0ec P 5786a53a9309463fbcda8fa320957ea9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:41:00.220058 4077 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 22:41:00.220207 4666 raft_consensus.cc:2258] T 3c629099418c49a3b3122c66cc6ff0ec P 5786a53a9309463fbcda8fa320957ea9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 22:41:00.220582 4077 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:41:00.220628 4667 ts_tablet_manager.cc:1905] T 3c629099418c49a3b3122c66cc6ff0ec P 2708f67c85c34e819ac68d960508e759: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:41:00.220990 4668 ts_tablet_manager.cc:1905] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:41:00.221669 4666 ts_tablet_manager.cc:1905] T 3c629099418c49a3b3122c66cc6ff0ec P 5786a53a9309463fbcda8fa320957ea9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:41:00.223390 4668 ts_tablet_manager.cc:1918] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:41:00.223490 4668 log.cc:1196] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-1/wal/wals/3c629099418c49a3b3122c66cc6ff0ec
I20260925 22:41:00.223526 4667 ts_tablet_manager.cc:1918] T 3c629099418c49a3b3122c66cc6ff0ec P 2708f67c85c34e819ac68d960508e759: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:41:00.223613 4667 log.cc:1196] T 3c629099418c49a3b3122c66cc6ff0ec P 2708f67c85c34e819ac68d960508e759: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-2/wal/wals/3c629099418c49a3b3122c66cc6ff0ec
I20260925 22:41:00.223820 4668 ts_tablet_manager.cc:1939] T 3c629099418c49a3b3122c66cc6ff0ec P 42cc1f16cdf64ad78464d04c1cf796a7: Deleting consensus metadata
I20260925 22:41:00.224488 4666 ts_tablet_manager.cc:1918] T 3c629099418c49a3b3122c66cc6ff0ec P 5786a53a9309463fbcda8fa320957ea9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:41:00.224567 4666 log.cc:1196] T 3c629099418c49a3b3122c66cc6ff0ec P 5786a53a9309463fbcda8fa320957ea9: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-0/wal/wals/3c629099418c49a3b3122c66cc6ff0ec
I20260925 22:41:00.224604 4064 catalog_manager.cc:4841] TS 42cc1f16cdf64ad78464d04c1cf796a7 (127.2.74.194:46367): tablet 3c629099418c49a3b3122c66cc6ff0ec (table test-table [id=88f4a0910f814a7fb05330bd96a8656f]) successfully deleted
I20260925 22:41:00.224875 4666 ts_tablet_manager.cc:1939] T 3c629099418c49a3b3122c66cc6ff0ec P 5786a53a9309463fbcda8fa320957ea9: Deleting consensus metadata
I20260925 22:41:00.225090 4667 ts_tablet_manager.cc:1939] T 3c629099418c49a3b3122c66cc6ff0ec P 2708f67c85c34e819ac68d960508e759: Deleting consensus metadata
I20260925 22:41:00.225795 4065 catalog_manager.cc:4841] TS 2708f67c85c34e819ac68d960508e759 (127.2.74.195:43423): tablet 3c629099418c49a3b3122c66cc6ff0ec (table test-table [id=88f4a0910f814a7fb05330bd96a8656f]) successfully deleted
I20260925 22:41:00.225890 4062 catalog_manager.cc:4841] TS 5786a53a9309463fbcda8fa320957ea9 (127.2.74.193:34679): tablet 3c629099418c49a3b3122c66cc6ff0ec (table test-table [id=88f4a0910f814a7fb05330bd96a8656f]) successfully deleted
I20260925 22:41:00.228017 4215 tablet_service.cc:1467] Processing CreateTablet for tablet a9d2811422614018a16ea93cfaea9533 (DEFAULT_TABLE table=test-table [id=df6de6313d3b4e6992b03bf9a878042a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:00.228040 4355 tablet_service.cc:1467] Processing CreateTablet for tablet a9d2811422614018a16ea93cfaea9533 (DEFAULT_TABLE table=test-table [id=df6de6313d3b4e6992b03bf9a878042a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:00.228116 4494 tablet_service.cc:1467] Processing CreateTablet for tablet a9d2811422614018a16ea93cfaea9533 (DEFAULT_TABLE table=test-table [id=df6de6313d3b4e6992b03bf9a878042a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:00.228343 4355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d2811422614018a16ea93cfaea9533. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:00.228338 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d2811422614018a16ea93cfaea9533. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:00.228353 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d2811422614018a16ea93cfaea9533. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:00.230533 4679 tablet_bootstrap.cc:492] T a9d2811422614018a16ea93cfaea9533 P 42cc1f16cdf64ad78464d04c1cf796a7: Bootstrap starting.
I20260925 22:41:00.230732 4677 tablet_bootstrap.cc:492] T a9d2811422614018a16ea93cfaea9533 P 5786a53a9309463fbcda8fa320957ea9: Bootstrap starting.
I20260925 22:41:00.230762 4678 tablet_bootstrap.cc:492] T a9d2811422614018a16ea93cfaea9533 P 2708f67c85c34e819ac68d960508e759: Bootstrap starting.
I20260925 22:41:00.231559 4678 tablet_bootstrap.cc:654] T a9d2811422614018a16ea93cfaea9533 P 2708f67c85c34e819ac68d960508e759: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:00.231559 4679 tablet_bootstrap.cc:654] T a9d2811422614018a16ea93cfaea9533 P 42cc1f16cdf64ad78464d04c1cf796a7: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:00.231693 4677 tablet_bootstrap.cc:654] T a9d2811422614018a16ea93cfaea9533 P 5786a53a9309463fbcda8fa320957ea9: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:00.232904 4678 tablet_bootstrap.cc:492] T a9d2811422614018a16ea93cfaea9533 P 2708f67c85c34e819ac68d960508e759: No bootstrap required, opened a new log
I20260925 22:41:00.232932 4677 tablet_bootstrap.cc:492] T a9d2811422614018a16ea93cfaea9533 P 5786a53a9309463fbcda8fa320957ea9: No bootstrap required, opened a new log
I20260925 22:41:00.233006 4678 ts_tablet_manager.cc:1397] T a9d2811422614018a16ea93cfaea9533 P 2708f67c85c34e819ac68d960508e759: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 22:41:00.233006 4677 ts_tablet_manager.cc:1397] T a9d2811422614018a16ea93cfaea9533 P 5786a53a9309463fbcda8fa320957ea9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:41:00.233443 4677 raft_consensus.cc:348] T a9d2811422614018a16ea93cfaea9533 P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:41:00.233443 4678 raft_consensus.cc:348] T a9d2811422614018a16ea93cfaea9533 P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:41:00.233563 4677 raft_consensus.cc:374] T a9d2811422614018a16ea93cfaea9533 P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:00.233577 4678 raft_consensus.cc:374] T a9d2811422614018a16ea93cfaea9533 P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:00.233601 4677 raft_consensus.cc:729] T a9d2811422614018a16ea93cfaea9533 P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5786a53a9309463fbcda8fa320957ea9, State: Initialized, Role: FOLLOWER
I20260925 22:41:00.233615 4678 raft_consensus.cc:729] T a9d2811422614018a16ea93cfaea9533 P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2708f67c85c34e819ac68d960508e759, State: Initialized, Role: FOLLOWER
I20260925 22:41:00.233743 4677 consensus_queue.cc:260] T a9d2811422614018a16ea93cfaea9533 P 5786a53a9309463fbcda8fa320957ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:41:00.233743 4678 consensus_queue.cc:260] T a9d2811422614018a16ea93cfaea9533 P 2708f67c85c34e819ac68d960508e759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:41:00.234010 4678 ts_tablet_manager.cc:1428] T a9d2811422614018a16ea93cfaea9533 P 2708f67c85c34e819ac68d960508e759: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:41:00.234011 4677 ts_tablet_manager.cc:1428] T a9d2811422614018a16ea93cfaea9533 P 5786a53a9309463fbcda8fa320957ea9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:41:00.235008 4679 tablet_bootstrap.cc:492] T a9d2811422614018a16ea93cfaea9533 P 42cc1f16cdf64ad78464d04c1cf796a7: No bootstrap required, opened a new log
I20260925 22:41:00.235116 4679 ts_tablet_manager.cc:1397] T a9d2811422614018a16ea93cfaea9533 P 42cc1f16cdf64ad78464d04c1cf796a7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 22:41:00.235658 4679 raft_consensus.cc:348] T a9d2811422614018a16ea93cfaea9533 P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:41:00.235811 4679 raft_consensus.cc:374] T a9d2811422614018a16ea93cfaea9533 P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:00.235852 4679 raft_consensus.cc:729] T a9d2811422614018a16ea93cfaea9533 P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cc1f16cdf64ad78464d04c1cf796a7, State: Initialized, Role: FOLLOWER
I20260925 22:41:00.236007 4679 consensus_queue.cc:260] T a9d2811422614018a16ea93cfaea9533 P 42cc1f16cdf64ad78464d04c1cf796a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:41:00.236316 4679 ts_tablet_manager.cc:1428] T a9d2811422614018a16ea93cfaea9533 P 42cc1f16cdf64ad78464d04c1cf796a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:41:00.432713 4683 raft_consensus.cc:482] T a9d2811422614018a16ea93cfaea9533 P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:41:00.432878 4683 raft_consensus.cc:504] T a9d2811422614018a16ea93cfaea9533 P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:41:00.433379 4683 leader_election.cc:288] T a9d2811422614018a16ea93cfaea9533 P 5786a53a9309463fbcda8fa320957ea9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cc1f16cdf64ad78464d04c1cf796a7 (127.2.74.194:46367), 2708f67c85c34e819ac68d960508e759 (127.2.74.195:43423)
I20260925 22:41:00.433876 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d2811422614018a16ea93cfaea9533" candidate_uuid: "5786a53a9309463fbcda8fa320957ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" is_pre_election: true
I20260925 22:41:00.433919 4515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d2811422614018a16ea93cfaea9533" candidate_uuid: "5786a53a9309463fbcda8fa320957ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2708f67c85c34e819ac68d960508e759" is_pre_election: true
I20260925 22:41:00.434053 4374 raft_consensus.cc:2454] T a9d2811422614018a16ea93cfaea9533 P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5786a53a9309463fbcda8fa320957ea9 in term 0.
I20260925 22:41:00.434118 4515 raft_consensus.cc:2454] T a9d2811422614018a16ea93cfaea9533 P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5786a53a9309463fbcda8fa320957ea9 in term 0.
I20260925 22:41:00.434533 4168 leader_election.cc:302] T a9d2811422614018a16ea93cfaea9533 P 5786a53a9309463fbcda8fa320957ea9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2708f67c85c34e819ac68d960508e759, 42cc1f16cdf64ad78464d04c1cf796a7, 5786a53a9309463fbcda8fa320957ea9; no voters:
I20260925 22:41:00.434772 4683 raft_consensus.cc:2781] T a9d2811422614018a16ea93cfaea9533 P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:41:00.434844 4683 raft_consensus.cc:482] T a9d2811422614018a16ea93cfaea9533 P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:41:00.434872 4683 raft_consensus.cc:3037] T a9d2811422614018a16ea93cfaea9533 P 5786a53a9309463fbcda8fa320957ea9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:00.436019 4683 raft_consensus.cc:504] T a9d2811422614018a16ea93cfaea9533 P 5786a53a9309463fbcda8fa320957ea9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:41:00.436434 4683 leader_election.cc:288] T a9d2811422614018a16ea93cfaea9533 P 5786a53a9309463fbcda8fa320957ea9 [CANDIDATE]: Term 1 election: Requested vote from peers 42cc1f16cdf64ad78464d04c1cf796a7 (127.2.74.194:46367), 2708f67c85c34e819ac68d960508e759 (127.2.74.195:43423)
I20260925 22:41:00.436956 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d2811422614018a16ea93cfaea9533" candidate_uuid: "5786a53a9309463fbcda8fa320957ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cc1f16cdf64ad78464d04c1cf796a7"
I20260925 22:41:00.436957 4515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d2811422614018a16ea93cfaea9533" candidate_uuid: "5786a53a9309463fbcda8fa320957ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2708f67c85c34e819ac68d960508e759"
I20260925 22:41:00.437100 4515 raft_consensus.cc:3037] T a9d2811422614018a16ea93cfaea9533 P 2708f67c85c34e819ac68d960508e759 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:00.437100 4374 raft_consensus.cc:3037] T a9d2811422614018a16ea93cfaea9533 P 42cc1f16cdf64ad78464d04c1cf796a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:00.438347 4515 raft_consensus.cc:2454] T a9d2811422614018a16ea93cfaea9533 P 2708f67c85c34e819ac68d960508e759 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5786a53a9309463fbcda8fa320957ea9 in term 1.
I20260925 22:41:00.438347 4374 raft_consensus.cc:2454] T a9d2811422614018a16ea93cfaea9533 P 42cc1f16cdf64ad78464d04c1cf796a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5786a53a9309463fbcda8fa320957ea9 in term 1.
I20260925 22:41:00.438731 4168 leader_election.cc:302] T a9d2811422614018a16ea93cfaea9533 P 5786a53a9309463fbcda8fa320957ea9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2708f67c85c34e819ac68d960508e759, 42cc1f16cdf64ad78464d04c1cf796a7, 5786a53a9309463fbcda8fa320957ea9; no voters:
I20260925 22:41:00.438912 4683 raft_consensus.cc:2781] T a9d2811422614018a16ea93cfaea9533 P 5786a53a9309463fbcda8fa320957ea9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:41:00.439155 4683 raft_consensus.cc:686] T a9d2811422614018a16ea93cfaea9533 P 5786a53a9309463fbcda8fa320957ea9 [term 1 LEADER]: Becoming Leader. State: Replica: 5786a53a9309463fbcda8fa320957ea9, State: Running, Role: LEADER
I20260925 22:41:00.439450 4683 consensus_queue.cc:237] T a9d2811422614018a16ea93cfaea9533 P 5786a53a9309463fbcda8fa320957ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } }
I20260925 22:41:00.442104 4078 catalog_manager.cc:5495] T a9d2811422614018a16ea93cfaea9533 P 5786a53a9309463fbcda8fa320957ea9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5786a53a9309463fbcda8fa320957ea9 (127.2.74.193). New cstate: current_term: 1 leader_uuid: "5786a53a9309463fbcda8fa320957ea9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5786a53a9309463fbcda8fa320957ea9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 34679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 } health_report { overall_health: UNKNOWN } } }
I20260925 22:41:00.484030 4695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:00.480387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38356 (local address 127.2.74.193:34679)
0925 22:41:00.480680 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:00.480688 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:00.480701 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:41:00.480787 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:00.480789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:00.480845 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:00.480969 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:41:00.481263 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:00.481797 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:00.482596 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:00.482829 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:00.482904 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:00.483172 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:41:00.483632 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:41:00.483730 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:00.483783 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 22:41:00.483885 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":137,"threads_started":1}
I20260925 22:41:00.494791 4698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:00.491484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:43423 (local address 127.2.74.193:44775)
0925 22:41:00.491820 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:00.491834 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:00.491921 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:00.492159 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:00.492161 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:00.492166 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:41:00.492277 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:00.492282 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:00.493006 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:00.493008 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:00.493663 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:00.493671 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:00.494110 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:00.494112 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:00.494185 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:00.494246 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:41:00.494529 (+ 283us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:41:00.494534 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:41:00.494584 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 22:41:00.494650 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":120,"threads_started":1}
I20260925 22:41:00.494781 4699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:00.491712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:44775 (local address 127.2.74.195:43423)
0925 22:41:00.491940 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:00.491956 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:00.491969 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:41:00.492015 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:00.492017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:00.492062 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:00.492137 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:41:00.492379 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:00.492906 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:00.493783 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:00.494003 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:00.494082 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:00.494370 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:41:00.494451 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:41:00.494506 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:00.494591 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 22:41:00.494638 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":81,"threads_started":1}
I20260925 22:41:00.501348 4515 raft_consensus.cc:1261] T a9d2811422614018a16ea93cfaea9533 P 2708f67c85c34e819ac68d960508e759 [term 1 FOLLOWER]: Refusing update from remote peer 5786a53a9309463fbcda8fa320957ea9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:41:00.501463 4374 raft_consensus.cc:1261] T a9d2811422614018a16ea93cfaea9533 P 42cc1f16cdf64ad78464d04c1cf796a7 [term 1 FOLLOWER]: Refusing update from remote peer 5786a53a9309463fbcda8fa320957ea9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:41:00.501878 4693 consensus_queue.cc:1035] T a9d2811422614018a16ea93cfaea9533 P 5786a53a9309463fbcda8fa320957ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2708f67c85c34e819ac68d960508e759" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:41:00.502120 4683 consensus_queue.cc:1035] T a9d2811422614018a16ea93cfaea9533 P 5786a53a9309463fbcda8fa320957ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cc1f16cdf64ad78464d04c1cf796a7" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 46367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:41:00.521974 4695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:00.517651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:44107 (local address 127.2.74.193:34679)
0925 22:41:00.517777 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:00.517783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:00.517960 (+ 177us) server_negotiation.cc:407] Connection header received
0925 22:41:00.518162 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:00.518165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:00.518226 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:00.518304 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:41:00.518584 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:00.519178 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:00.520891 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:00.521137 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:00.521233 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:00.521529 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:41:00.521616 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:41:00.521693 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:00.521801 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 22:41:00.521853 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:41:00.522039 4704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:00.517583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:34679 (local address 127.2.74.195:44107)
0925 22:41:00.517851 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:00.517872 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:00.518035 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:00.518336 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:00.518339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:00.518349 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:41:00.518466 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:00.518472 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:00.519290 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:00.519293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:00.520746 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:00.520757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:00.521250 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:00.521254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:00.521354 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:00.521389 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:41:00.521716 (+ 327us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:41:00.521722 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:41:00.521795 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 22:41:00.521888 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":121,"threads_started":1}
W20260925 22:41:00.522871 4648 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 22:41:02.089293 4076 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:41:02.089449 4076 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:41:02.092057 4076 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b92fc40cacb846a1990f43fa527acdcb: Sending DeleteTablet for 3 replicas of tablet a9d2811422614018a16ea93cfaea9533
I20260925 22:41:02.092680 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet a9d2811422614018a16ea93cfaea9533 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:02 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:38346
I20260925 22:41:02.092731 4355 tablet_service.cc:1514] Processing DeleteTablet for tablet a9d2811422614018a16ea93cfaea9533 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:02 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:42180
I20260925 22:41:02.092787 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet a9d2811422614018a16ea93cfaea9533 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:02 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:49146
I20260925 22:41:02.093107 4729 tablet_replica.cc:330] T a9d2811422614018a16ea93cfaea9533 P 2708f67c85c34e819ac68d960508e759: stopping tablet replica
I20260925 22:41:02.093237 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 4150
I20260925 22:41:02.093251 4729 raft_consensus.cc:2229] T a9d2811422614018a16ea93cfaea9533 P 2708f67c85c34e819ac68d960508e759 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:41:02.094183 4729 raft_consensus.cc:2258] T a9d2811422614018a16ea93cfaea9533 P 2708f67c85c34e819ac68d960508e759 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:41:02.094462 4727 tablet_replica.cc:330] T a9d2811422614018a16ea93cfaea9533 P 42cc1f16cdf64ad78464d04c1cf796a7: stopping tablet replica
I20260925 22:41:02.094615 4727 raft_consensus.cc:2229] T a9d2811422614018a16ea93cfaea9533 P 42cc1f16cdf64ad78464d04c1cf796a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:41:02.094750 4727 raft_consensus.cc:2258] T a9d2811422614018a16ea93cfaea9533 P 42cc1f16cdf64ad78464d04c1cf796a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:41:02.095266 4729 ts_tablet_manager.cc:1905] T a9d2811422614018a16ea93cfaea9533 P 2708f67c85c34e819ac68d960508e759: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:41:02.095435 4727 ts_tablet_manager.cc:1905] T a9d2811422614018a16ea93cfaea9533 P 42cc1f16cdf64ad78464d04c1cf796a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:41:02.097625 4727 ts_tablet_manager.cc:1918] T a9d2811422614018a16ea93cfaea9533 P 42cc1f16cdf64ad78464d04c1cf796a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:41:02.097694 4727 log.cc:1196] T a9d2811422614018a16ea93cfaea9533 P 42cc1f16cdf64ad78464d04c1cf796a7: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-1/wal/wals/a9d2811422614018a16ea93cfaea9533
I20260925 22:41:02.097898 4727 ts_tablet_manager.cc:1939] T a9d2811422614018a16ea93cfaea9533 P 42cc1f16cdf64ad78464d04c1cf796a7: Deleting consensus metadata
I20260925 22:41:02.098675 4064 catalog_manager.cc:4841] TS 42cc1f16cdf64ad78464d04c1cf796a7 (127.2.74.194:46367): tablet a9d2811422614018a16ea93cfaea9533 (table test-table [id=df6de6313d3b4e6992b03bf9a878042a]) successfully deleted
I20260925 22:41:02.098722 4729 ts_tablet_manager.cc:1918] T a9d2811422614018a16ea93cfaea9533 P 2708f67c85c34e819ac68d960508e759: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:41:02.098799 4729 log.cc:1196] T a9d2811422614018a16ea93cfaea9533 P 2708f67c85c34e819ac68d960508e759: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376047580784-2347-0/minicluster-data/ts-2/wal/wals/a9d2811422614018a16ea93cfaea9533
I20260925 22:41:02.099092 4729 ts_tablet_manager.cc:1939] T a9d2811422614018a16ea93cfaea9533 P 2708f67c85c34e819ac68d960508e759: Deleting consensus metadata
I20260925 22:41:02.099862 4065 catalog_manager.cc:4841] TS 2708f67c85c34e819ac68d960508e759 (127.2.74.195:43423): tablet a9d2811422614018a16ea93cfaea9533 (table test-table [id=df6de6313d3b4e6992b03bf9a878042a]) successfully deleted
W20260925 22:41:02.101822 4062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.74.193:34679 (error 108)
I20260925 22:41:02.102021 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 4290
I20260925 22:41:02.104321 4731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:02.103905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:34679 (local address 127.0.0.1:38366)
0925 22:41:02.104164 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:02.104235 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.193:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":105,"threads_started":1}
W20260925 22:41:02.104462 4062 catalog_manager.cc:4584] TS 5786a53a9309463fbcda8fa320957ea9 (127.2.74.193:34679): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a9d2811422614018a16ea93cfaea9533: Network error: Client connection negotiation failed: client connection to 127.2.74.193:34679: connect: Connection refused (error 111)
I20260925 22:41:02.108855 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 4430
I20260925 22:41:02.115664 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 4046
2026-09-25T22:41:02Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6706 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:41:02 dist-test-slave-csnl krb5kdc[4734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:41:02 dist-test-slave-csnl krb5kdc[4734](info): set up 2 sockets
Sep 25 22:41:02 dist-test-slave-csnl krb5kdc[4734](info): commencing operation
krb5kdc: starting...
W20260925 22:41:03.183215 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:41:03 dist-test-slave-csnl krb5kdc[4734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:41:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:41:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:41:03.308233 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:39805
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:39805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:03.427166 4747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:03.427500 4747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:03.427543 4747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:03.432096 4747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:41:03.432207 4747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:03.432230 4747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:03.432248 4747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:41:03.432264 4747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:41:03.438799 4747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:39805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:39805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:03.440109 4747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:03.441200 4747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:03.451057 4755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:03.451119 4747 server_base.cc:1034] running on GCE node
W20260925 22:41:03.451063 4753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:03.451061 4752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:03.452046 4747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:03.453281 4747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:03.454506 4747 hybrid_clock.cc:648] HybridClock initialized: now 1790376063454480 us; error 45 us; skew 500 ppm
Sep 25 22:41:03 dist-test-slave-csnl krb5kdc[4734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376063, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:03.457950 4747 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:41:03.458937 4747 webserver.cc:460] Webserver started at http://127.2.74.254:34933/ using document root <none> and password file <none>
I20260925 22:41:03.459617 4747 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:03.459715 4747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:03.459960 4747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:03.462262 4747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "f497f3e083eb41c88ed7319f0e0fdbf8"
format_stamp: "Formatted at 2026-09-25 22:41:03 on dist-test-slave-csnl"
server_key: "c5835192eb8b24c7f7515e2c9526ae5d"
server_key_iv: "eced1b3a718bdc6ce154c1efede7221a"
server_key_version: "encryptionkey@0"
I20260925 22:41:03.462920 4747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "f497f3e083eb41c88ed7319f0e0fdbf8"
format_stamp: "Formatted at 2026-09-25 22:41:03 on dist-test-slave-csnl"
server_key: "c5835192eb8b24c7f7515e2c9526ae5d"
server_key_iv: "eced1b3a718bdc6ce154c1efede7221a"
server_key_version: "encryptionkey@0"
I20260925 22:41:03.467296 4747 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:41:03.470278 4762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:03.471812 4747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:41:03.471993 4747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "f497f3e083eb41c88ed7319f0e0fdbf8"
format_stamp: "Formatted at 2026-09-25 22:41:03 on dist-test-slave-csnl"
server_key: "c5835192eb8b24c7f7515e2c9526ae5d"
server_key_iv: "eced1b3a718bdc6ce154c1efede7221a"
server_key_version: "encryptionkey@0"
I20260925 22:41:03.472132 4747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:03.506479 4747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:03.509766 4747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:03.510010 4747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:03.517656 4747 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:39805
I20260925 22:41:03.517675 4814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:39805 every 8 connection(s)
I20260925 22:41:03.518726 4747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:41:03.521647 4815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:03.525869 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 4747
I20260925 22:41:03.525980 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:41:03.526248 2347 external_mini_cluster.cc:1428] Setting key efa97bb8c1a10eeddd7b7406bf0c8477
I20260925 22:41:03.528296 4815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f497f3e083eb41c88ed7319f0e0fdbf8: Bootstrap starting.
I20260925 22:41:03.530711 4815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f497f3e083eb41c88ed7319f0e0fdbf8: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:03.531627 4815 log.cc:824] T 00000000000000000000000000000000 P f497f3e083eb41c88ed7319f0e0fdbf8: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:03.533775 4815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f497f3e083eb41c88ed7319f0e0fdbf8: No bootstrap required, opened a new log
Sep 25 22:41:03 dist-test-slave-csnl krb5kdc[4734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:41:03.538161 4815 raft_consensus.cc:348] T 00000000000000000000000000000000 P f497f3e083eb41c88ed7319f0e0fdbf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f497f3e083eb41c88ed7319f0e0fdbf8" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 39805 } }
I20260925 22:41:03.538470 4815 raft_consensus.cc:374] T 00000000000000000000000000000000 P f497f3e083eb41c88ed7319f0e0fdbf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:03.538555 4815 raft_consensus.cc:729] T 00000000000000000000000000000000 P f497f3e083eb41c88ed7319f0e0fdbf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f497f3e083eb41c88ed7319f0e0fdbf8, State: Initialized, Role: FOLLOWER
I20260925 22:41:03.539125 4815 consensus_queue.cc:260] T 00000000000000000000000000000000 P f497f3e083eb41c88ed7319f0e0fdbf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f497f3e083eb41c88ed7319f0e0fdbf8" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 39805 } }
I20260925 22:41:03.539307 4815 raft_consensus.cc:388] T 00000000000000000000000000000000 P f497f3e083eb41c88ed7319f0e0fdbf8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:41:03.539377 4815 raft_consensus.cc:482] T 00000000000000000000000000000000 P f497f3e083eb41c88ed7319f0e0fdbf8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:41:03.539534 4815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f497f3e083eb41c88ed7319f0e0fdbf8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:03.540522 4815 raft_consensus.cc:504] T 00000000000000000000000000000000 P f497f3e083eb41c88ed7319f0e0fdbf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f497f3e083eb41c88ed7319f0e0fdbf8" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 39805 } }
I20260925 22:41:03.540901 4815 leader_election.cc:302] T 00000000000000000000000000000000 P f497f3e083eb41c88ed7319f0e0fdbf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f497f3e083eb41c88ed7319f0e0fdbf8; no voters:
I20260925 22:41:03.541337 4815 leader_election.cc:288] T 00000000000000000000000000000000 P f497f3e083eb41c88ed7319f0e0fdbf8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:41:03.541415 4820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f497f3e083eb41c88ed7319f0e0fdbf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:41:03.541715 4820 raft_consensus.cc:686] T 00000000000000000000000000000000 P f497f3e083eb41c88ed7319f0e0fdbf8 [term 1 LEADER]: Becoming Leader. State: Replica: f497f3e083eb41c88ed7319f0e0fdbf8, State: Running, Role: LEADER
I20260925 22:41:03.542092 4820 consensus_queue.cc:237] T 00000000000000000000000000000000 P f497f3e083eb41c88ed7319f0e0fdbf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f497f3e083eb41c88ed7319f0e0fdbf8" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 39805 } }
I20260925 22:41:03.542870 4815 sys_catalog.cc:564] T 00000000000000000000000000000000 P f497f3e083eb41c88ed7319f0e0fdbf8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:41:03.542923 4818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:03.527843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58238 (local address 127.2.74.254:39805)
0925 22:41:03.528312 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:03.528329 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:03.528368 (+ 39us) server_negotiation.cc:407] Connection header received
0925 22:41:03.529115 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:03.529133 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:03.529508 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:03.529902 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:03.530758 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:03.531994 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:03.532934 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:03.533463 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:03.533657 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:03.536601 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:03.536608 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:03.536625 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:03.536662 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:03.539325 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:03.539916 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:03.539924 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:03.539931 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:03.540066 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:03.540386 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:03.540389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:03.540390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:03.540901 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:03.541128 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:03.541443 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 22:41:03.541978 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":152,"threads_started":1}
I20260925 22:41:03.543432 4820 sys_catalog.cc:455] T 00000000000000000000000000000000 P f497f3e083eb41c88ed7319f0e0fdbf8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f497f3e083eb41c88ed7319f0e0fdbf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f497f3e083eb41c88ed7319f0e0fdbf8" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 39805 } } }
I20260925 22:41:03.543557 4820 sys_catalog.cc:458] T 00000000000000000000000000000000 P f497f3e083eb41c88ed7319f0e0fdbf8 [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:03.543838 4821 sys_catalog.cc:455] T 00000000000000000000000000000000 P f497f3e083eb41c88ed7319f0e0fdbf8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f497f3e083eb41c88ed7319f0e0fdbf8. Latest consensus state: current_term: 1 leader_uuid: "f497f3e083eb41c88ed7319f0e0fdbf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f497f3e083eb41c88ed7319f0e0fdbf8" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 39805 } } }
I20260925 22:41:03.543929 4821 sys_catalog.cc:458] T 00000000000000000000000000000000 P f497f3e083eb41c88ed7319f0e0fdbf8 [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:03.545746 4824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:41:03.548635 4824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:41:03.555135 4824 catalog_manager.cc:1347] Generated new cluster ID: b70c2aba11f846388b28f3ddfa3f43ee
I20260925 22:41:03.555231 4824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:41:03.571544 4824 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:41:03.572608 4824 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:41:03.581025 4824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f497f3e083eb41c88ed7319f0e0fdbf8: Generated new TSK 0
I20260925 22:41:03.581843 4824 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
I20260925 22:41:03.616389 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:0
--local_ip_for_outbound_sockets=127.2.74.193
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:39805
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:03.734100 4841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:03.734411 4841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:03.734458 4841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:03.738770 4841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:03.738864 4841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:03.738927 4841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:41:03.744880 4841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:39805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:03.746045 4841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:03.747159 4841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:41:03.757486 4841 server_base.cc:1034] running on GCE node
W20260925 22:41:03.757400 4847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:03.757433 4846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:03.757412 4849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:03.758196 4841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:03.758925 4841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:03.760156 4841 hybrid_clock.cc:648] HybridClock initialized: now 1790376063760141 us; error 32 us; skew 500 ppm
Sep 25 22:41:03 dist-test-slave-csnl krb5kdc[4734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376063, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:03.763161 4841 init.cc:377] Logged in from keytab as kudu/127.2.74.193@KRBTEST.COM (short username kudu)
I20260925 22:41:03.764122 4841 webserver.cc:460] Webserver started at http://127.2.74.193:33445/ using document root <none> and password file <none>
I20260925 22:41:03.764719 4841 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:03.764784 4841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:03.765040 4841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:03.767302 4841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/data/instance:
uuid: "8fbff717acdf4ee1b69f391843db4be8"
format_stamp: "Formatted at 2026-09-25 22:41:03 on dist-test-slave-csnl"
server_key: "eb6eded6c57d83572eb891d05c74eb79"
server_key_iv: "59b4794da23a4c63fd79d84506411fc9"
server_key_version: "encryptionkey@0"
I20260925 22:41:03.767933 4841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance:
uuid: "8fbff717acdf4ee1b69f391843db4be8"
format_stamp: "Formatted at 2026-09-25 22:41:03 on dist-test-slave-csnl"
server_key: "eb6eded6c57d83572eb891d05c74eb79"
server_key_iv: "59b4794da23a4c63fd79d84506411fc9"
server_key_version: "encryptionkey@0"
I20260925 22:41:03.771952 4841 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260925 22:41:03.774613 4856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:03.776131 4841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:41:03.776280 4841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "8fbff717acdf4ee1b69f391843db4be8"
format_stamp: "Formatted at 2026-09-25 22:41:03 on dist-test-slave-csnl"
server_key: "eb6eded6c57d83572eb891d05c74eb79"
server_key_iv: "59b4794da23a4c63fd79d84506411fc9"
server_key_version: "encryptionkey@0"
I20260925 22:41:03.776415 4841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:03.809623 4841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:03.813179 4841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:03.813409 4841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:03.814069 4841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:03.814971 4841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:03.815018 4841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:03.815088 4841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:03.815121 4841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:03.824218 4841 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:46203
I20260925 22:41:03.824235 4969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:46203 every 8 connection(s)
I20260925 22:41:03.825289 4841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:41:03.834149 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 4841
I20260925 22:41:03.834302 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance
I20260925 22:41:03.834606 2347 external_mini_cluster.cc:1428] Setting key c144f4fcef57a97d0492bbfa765ec153
Sep 25 22:41:03 dist-test-slave-csnl krb5kdc[4734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376063, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:41:03.840390 4818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:03.827204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:52221 (local address 127.2.74.254:39805)
0925 22:41:03.827352 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:03.827355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:03.828266 (+ 911us) server_negotiation.cc:407] Connection header received
0925 22:41:03.829672 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:03.829675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:03.829732 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:03.829843 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:03.831166 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:03.831736 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:03.833065 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:03.833311 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:03.833429 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:03.836244 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:03.836248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:03.836250 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:03.836273 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:03.837871 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:03.838457 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:03.838463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:03.838465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:03.838565 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:03.839017 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:03.839023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:03.839028 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:03.839316 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:03.839422 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:03.840005 (+ 583us) server_negotiation.cc:299] Negotiation successful
0925 22:41:03.840173 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:41:03.841223 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:03.827615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:39805 (local address 127.2.74.193:52221)
0925 22:41:03.828106 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:03.828152 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:03.829440 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:03.830027 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:03.830034 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:03.830422 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:03.830939 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:03.831020 (+ 81us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:03.831881 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:03.831884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:03.832924 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:03.832936 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:03.833474 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:03.833478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:03.833667 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:03.834337 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:03.834368 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:03.836057 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:03.838014 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:03.838020 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:03.838034 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:03.838335 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:03.838699 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:03.838701 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:03.838702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:03.838869 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:03.839450 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:03.839456 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:03.839729 (+ 273us) client_negotiation.cc:769] Sending connection context
0925 22:41:03.839900 (+ 171us) client_negotiation.cc:240] Negotiation successful
0925 22:41:03.840188 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":124,"threads_started":1}
I20260925 22:41:03.842531 4970 heartbeater.cc:347] Connected to a master server at 127.2.74.254:39805
I20260925 22:41:03.842778 4970 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:03.843407 4970 heartbeater.cc:511] Master 127.2.74.254:39805 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.74.194@KRBTEST.COM; defaulting to no policy
I20260925 22:41:03.845134 4779 ts_manager.cc:194] Registered new tserver with Master: 8fbff717acdf4ee1b69f391843db4be8 (127.2.74.193:46203)
I20260925 22:41:03.846356 4779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.193@KRBTEST.COM'} at 127.2.74.193:52221
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
I20260925 22:41:03.862227 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:0
--local_ip_for_outbound_sockets=127.2.74.194
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:39805
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:03.979380 4975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:03.979688 4975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:03.979735 4975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:03.984306 4975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:03.984396 4975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:03.984462 4975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:41:03.990319 4975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:39805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:03.991629 4975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:03.992760 4975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:04.003118 4980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:04.003116 4981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:04.003178 4983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:04.003593 4975 server_base.cc:1034] running on GCE node
I20260925 22:41:04.004081 4975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:04.004798 4975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:04.006021 4975 hybrid_clock.cc:648] HybridClock initialized: now 1790376064006000 us; error 38 us; skew 500 ppm
Sep 25 22:41:04 dist-test-slave-csnl krb5kdc[4734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376064, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:04.009107 4975 init.cc:377] Logged in from keytab as kudu/127.2.74.194@KRBTEST.COM (short username kudu)
I20260925 22:41:04.010070 4975 webserver.cc:460] Webserver started at http://127.2.74.194:46679/ using document root <none> and password file <none>
I20260925 22:41:04.010648 4975 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:04.010716 4975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:04.010896 4975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:04.013226 4975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/data/instance:
uuid: "510036101caa491484667cefa79d1410"
format_stamp: "Formatted at 2026-09-25 22:41:04 on dist-test-slave-csnl"
server_key: "2620d4b402a7d04c5cf95c8cacc8cad9"
server_key_iv: "d84866e5eb74321d3e82744cc2a025a2"
server_key_version: "encryptionkey@0"
I20260925 22:41:04.013784 4975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance:
uuid: "510036101caa491484667cefa79d1410"
format_stamp: "Formatted at 2026-09-25 22:41:04 on dist-test-slave-csnl"
server_key: "2620d4b402a7d04c5cf95c8cacc8cad9"
server_key_iv: "d84866e5eb74321d3e82744cc2a025a2"
server_key_version: "encryptionkey@0"
I20260925 22:41:04.017863 4975 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:41:04.020900 4990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:04.022351 4975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:41:04.022481 4975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "510036101caa491484667cefa79d1410"
format_stamp: "Formatted at 2026-09-25 22:41:04 on dist-test-slave-csnl"
server_key: "2620d4b402a7d04c5cf95c8cacc8cad9"
server_key_iv: "d84866e5eb74321d3e82744cc2a025a2"
server_key_version: "encryptionkey@0"
I20260925 22:41:04.022596 4975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:04.042265 4975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:04.046180 4975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:04.046355 4975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:04.046942 4975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:04.047941 4975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:04.047999 4975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:04.048089 4975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:04.048132 4975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:04.057667 4975 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:36131
I20260925 22:41:04.057689 5103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:36131 every 8 connection(s)
I20260925 22:41:04.058698 4975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:41:04.059618 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 4975
I20260925 22:41:04.059739 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance
I20260925 22:41:04.059990 2347 external_mini_cluster.cc:1428] Setting key 0c0afe9e288dfa6676d376a686e2e0f3
WARNING: no policy specified for kudu/127.2.74.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.195@KRBTEST.COM" created.
Sep 25 22:41:04 dist-test-slave-csnl krb5kdc[4734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376064, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:41:04.077814 4818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:04.061745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:51445 (local address 127.2.74.254:39805)
0925 22:41:04.061875 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:04.061879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:04.063182 (+ 1303us) server_negotiation.cc:407] Connection header received
0925 22:41:04.065265 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:04.065269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:04.065333 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:04.065415 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:04.066741 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:04.067370 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:04.068606 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:04.068836 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:04.068956 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:04.073326 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:04.073329 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:04.073332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:04.073360 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:04.075300 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:04.076023 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:04.076027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:04.076029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:04.076115 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:04.076487 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:04.076492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:04.076494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:04.076800 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:04.076897 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:04.077495 (+ 598us) server_negotiation.cc:299] Negotiation successful
0925 22:41:04.077647 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:41:04.078758 5106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:04.062374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:39805 (local address 127.2.74.194:51445)
0925 22:41:04.063015 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:04.063069 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:04.064986 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:04.065611 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:04.065621 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:04.066022 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:04.066565 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:04.066580 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:04.067513 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:04.067517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:04.068454 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:04.068464 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:04.069008 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:04.069012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:04.069204 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:04.069864 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:04.069883 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:04.073086 (+ 3203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:04.075455 (+ 2369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:04.075469 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:04.075507 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:04.075925 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:04.076216 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:04.076220 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:04.076222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:04.076386 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:04.076898 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:04.076907 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:04.077238 (+ 331us) client_negotiation.cc:769] Sending connection context
0925 22:41:04.077454 (+ 216us) client_negotiation.cc:240] Negotiation successful
0925 22:41:04.077824 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":167,"threads_started":1}
I20260925 22:41:04.080274 5104 heartbeater.cc:347] Connected to a master server at 127.2.74.254:39805
I20260925 22:41:04.080582 5104 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:04.081275 5104 heartbeater.cc:511] Master 127.2.74.254:39805 requested a full tablet report, sending...
I20260925 22:41:04.082532 4779 ts_manager.cc:194] Registered new tserver with Master: 510036101caa491484667cefa79d1410 (127.2.74.194:36131)
I20260925 22:41:04.083287 4779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.194@KRBTEST.COM'} at 127.2.74.194:51445
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
I20260925 22:41:04.089885 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:0
--local_ip_for_outbound_sockets=127.2.74.195
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:39805
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:04.209007 5109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:04.209313 5109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:04.209357 5109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:04.213642 5109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:04.213734 5109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:04.213797 5109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:41:04.219517 5109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:39805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:04.220602 5109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:04.221638 5109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:04.231738 5115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:04.231793 5109 server_base.cc:1034] running on GCE node
W20260925 22:41:04.231755 5114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:04.231753 5117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:04.232362 5109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:04.233006 5109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:04.234197 5109 hybrid_clock.cc:648] HybridClock initialized: now 1790376064234182 us; error 28 us; skew 500 ppm
Sep 25 22:41:04 dist-test-slave-csnl krb5kdc[4734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376064, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:04.236979 5109 init.cc:377] Logged in from keytab as kudu/127.2.74.195@KRBTEST.COM (short username kudu)
I20260925 22:41:04.237884 5109 webserver.cc:460] Webserver started at http://127.2.74.195:33189/ using document root <none> and password file <none>
I20260925 22:41:04.238464 5109 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:04.238557 5109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:04.238794 5109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:04.241021 5109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/data/instance:
uuid: "a31162b4db5a418da12c462bb2200684"
format_stamp: "Formatted at 2026-09-25 22:41:04 on dist-test-slave-csnl"
server_key: "db6f37f48a1244939f055796c924ff5c"
server_key_iv: "30c46482e26646593d7a8d8a6a5bc4d2"
server_key_version: "encryptionkey@0"
I20260925 22:41:04.241631 5109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance:
uuid: "a31162b4db5a418da12c462bb2200684"
format_stamp: "Formatted at 2026-09-25 22:41:04 on dist-test-slave-csnl"
server_key: "db6f37f48a1244939f055796c924ff5c"
server_key_iv: "30c46482e26646593d7a8d8a6a5bc4d2"
server_key_version: "encryptionkey@0"
I20260925 22:41:04.245810 5109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 22:41:04.248500 5124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:04.249890 5109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:41:04.250051 5109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "a31162b4db5a418da12c462bb2200684"
format_stamp: "Formatted at 2026-09-25 22:41:04 on dist-test-slave-csnl"
server_key: "db6f37f48a1244939f055796c924ff5c"
server_key_iv: "30c46482e26646593d7a8d8a6a5bc4d2"
server_key_version: "encryptionkey@0"
I20260925 22:41:04.250187 5109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:04.259964 5109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:04.262941 5109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:04.263226 5109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:04.263839 5109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:04.264714 5109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:04.264787 5109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:04.264873 5109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:04.264922 5109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:04.273890 5109 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:37373
I20260925 22:41:04.273918 5237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:37373 every 8 connection(s)
I20260925 22:41:04.274909 5109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:41:04.276628 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 5109
I20260925 22:41:04.276767 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance
I20260925 22:41:04.277088 2347 external_mini_cluster.cc:1428] Setting key f1451ddea0386eb9b52f7dbce30ed576
Sep 25 22:41:04 dist-test-slave-csnl krb5kdc[4734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376064, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:41:04.290138 4818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:04.277131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:60037 (local address 127.2.74.254:39805)
0925 22:41:04.277297 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:04.277302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:04.278493 (+ 1191us) server_negotiation.cc:407] Connection header received
0925 22:41:04.280199 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:04.280203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:04.280261 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:04.280340 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:04.281699 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:04.282243 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:04.283551 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:04.283786 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:04.283904 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:04.286534 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:04.286537 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:04.286539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:04.286565 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:04.287908 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:04.288471 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:04.288475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:04.288477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:04.288561 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:04.288913 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:04.288918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:04.288920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:04.289112 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:04.289222 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:04.289806 (+ 584us) server_negotiation.cc:299] Negotiation successful
0925 22:41:04.289965 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 22:41:04.291237 5240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:04.277743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:39805 (local address 127.2.74.195:60037)
0925 22:41:04.278345 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:04.278393 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:04.279981 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:04.280524 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:04.280532 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:04.280984 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:04.281539 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:04.281554 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:04.282396 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:04.282400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:04.283413 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:04.283423 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:04.283955 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:04.283959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:04.284137 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:04.284735 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:04.284753 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:04.286321 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:04.288038 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:04.288045 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:04.288064 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:04.288359 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:04.288657 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:04.288661 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:04.288663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:04.288818 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:04.289223 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:04.289240 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:04.289506 (+ 266us) client_negotiation.cc:769] Sending connection context
0925 22:41:04.289741 (+ 235us) client_negotiation.cc:240] Negotiation successful
0925 22:41:04.290090 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":159,"threads_started":1}
I20260925 22:41:04.292264 5238 heartbeater.cc:347] Connected to a master server at 127.2.74.254:39805
I20260925 22:41:04.292491 5238 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:04.292949 5238 heartbeater.cc:511] Master 127.2.74.254:39805 requested a full tablet report, sending...
I20260925 22:41:04.293947 4779 ts_manager.cc:194] Registered new tserver with Master: a31162b4db5a418da12c462bb2200684 (127.2.74.195:37373)
I20260925 22:41:04.294490 4779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.195@KRBTEST.COM'} at 127.2.74.195:60037
I20260925 22:41:04.303344 2347 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 22:41:04.316535 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:04.315061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58240 (local address 127.2.74.254:39805)
0925 22:41:04.315219 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:04.315225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:04.315378 (+ 153us) server_negotiation.cc:407] Connection header received
0925 22:41:04.315498 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:04.315501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:04.315547 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:04.315619 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:04.316386 (+ 767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58240: BlockingRecv error: recv got EOF from 127.0.0.1:58240 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 22:41:04.317660 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 4841
I20260925 22:41:04.323158 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 4975
I20260925 22:41:04.328694 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 5109
I20260925 22:41:04.334571 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 4747
2026-09-25T22:41:04Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2217 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:41:04 dist-test-slave-csnl krb5kdc[5250](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:41:04 dist-test-slave-csnl krb5kdc[5250](info): set up 2 sockets
Sep 25 22:41:04 dist-test-slave-csnl krb5kdc[5250](info): commencing operation
krb5kdc: starting...
W20260925 22:41:05.375643 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:41:05 dist-test-slave-csnl krb5kdc[5250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:41:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:41:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:41:05.501631 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:36331
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:36331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:05.616282 5263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:05.616655 5263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:05.616740 5263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:05.621176 5263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:41:05.621299 5263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:05.621335 5263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:05.621379 5263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:41:05.621416 5263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:41:05.627183 5263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:36331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:36331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:05.628407 5263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:05.629504 5263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:05.638406 5268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:05.638458 5271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:05.638489 5269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:05.638707 5263 server_base.cc:1034] running on GCE node
I20260925 22:41:05.639441 5263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:05.640615 5263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:05.641776 5263 hybrid_clock.cc:648] HybridClock initialized: now 1790376065641764 us; error 28 us; skew 500 ppm
Sep 25 22:41:05 dist-test-slave-csnl krb5kdc[5250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376065, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:05.644737 5263 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:41:05.645756 5263 webserver.cc:460] Webserver started at http://127.2.74.254:34521/ using document root <none> and password file <none>
I20260925 22:41:05.646341 5263 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:05.646435 5263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:05.646669 5263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:05.648887 5263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "1c644b1536f34cbf9b7ce26eaef34710"
format_stamp: "Formatted at 2026-09-25 22:41:05 on dist-test-slave-csnl"
server_key: "b6396a60b553a550eff39d1ae92214e1"
server_key_iv: "8dd8c829477325d0ead0f8748fd2b6c2"
server_key_version: "encryptionkey@0"
I20260925 22:41:05.649495 5263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "1c644b1536f34cbf9b7ce26eaef34710"
format_stamp: "Formatted at 2026-09-25 22:41:05 on dist-test-slave-csnl"
server_key: "b6396a60b553a550eff39d1ae92214e1"
server_key_iv: "8dd8c829477325d0ead0f8748fd2b6c2"
server_key_version: "encryptionkey@0"
I20260925 22:41:05.653565 5263 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:41:05.656198 5278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:05.657533 5263 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:41:05.657688 5263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "1c644b1536f34cbf9b7ce26eaef34710"
format_stamp: "Formatted at 2026-09-25 22:41:05 on dist-test-slave-csnl"
server_key: "b6396a60b553a550eff39d1ae92214e1"
server_key_iv: "8dd8c829477325d0ead0f8748fd2b6c2"
server_key_version: "encryptionkey@0"
I20260925 22:41:05.657822 5263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:05.674782 5263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:05.677766 5263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:05.677983 5263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:05.685237 5263 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:36331
I20260925 22:41:05.685549 5330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:36331 every 8 connection(s)
I20260925 22:41:05.686273 5263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:41:05.688405 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 5263
I20260925 22:41:05.688534 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:41:05.688818 2347 external_mini_cluster.cc:1428] Setting key 9c13404a9f798f7ac5d9b730c3083ecb
I20260925 22:41:05.689582 5331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:05.696419 5331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c644b1536f34cbf9b7ce26eaef34710: Bootstrap starting.
Sep 25 22:41:05 dist-test-slave-csnl krb5kdc[5250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:41:05.699331 5331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c644b1536f34cbf9b7ce26eaef34710: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:05.700302 5331 log.cc:824] T 00000000000000000000000000000000 P 1c644b1536f34cbf9b7ce26eaef34710: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:05.702989 5331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c644b1536f34cbf9b7ce26eaef34710: No bootstrap required, opened a new log
I20260925 22:41:05.704056 5334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:05.690131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54946 (local address 127.2.74.254:36331)
0925 22:41:05.690561 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:05.690584 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:05.690619 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:41:05.691349 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:05.691368 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:05.691706 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:05.691990 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:05.692939 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:05.693920 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:05.695148 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:05.695609 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:05.695736 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:05.698546 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:05.698553 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:05.698570 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:05.698605 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:05.700754 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:05.701351 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:05.701357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:05.701365 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:05.701502 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:05.701776 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:05.701779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:05.701780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:05.702308 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:05.702511 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:05.702718 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 22:41:05.703062 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":144,"threads_started":1}
I20260925 22:41:05.706312 5331 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c644b1536f34cbf9b7ce26eaef34710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c644b1536f34cbf9b7ce26eaef34710" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36331 } }
I20260925 22:41:05.706640 5331 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c644b1536f34cbf9b7ce26eaef34710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:05.706702 5331 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c644b1536f34cbf9b7ce26eaef34710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c644b1536f34cbf9b7ce26eaef34710, State: Initialized, Role: FOLLOWER
I20260925 22:41:05.707230 5331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c644b1536f34cbf9b7ce26eaef34710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c644b1536f34cbf9b7ce26eaef34710" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36331 } }
I20260925 22:41:05.707396 5331 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c644b1536f34cbf9b7ce26eaef34710 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:41:05.707477 5331 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c644b1536f34cbf9b7ce26eaef34710 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:41:05.707553 2347 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:41:05.707584 5331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c644b1536f34cbf9b7ce26eaef34710 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:05.708565 5331 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c644b1536f34cbf9b7ce26eaef34710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c644b1536f34cbf9b7ce26eaef34710" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36331 } }
I20260925 22:41:05.708941 5331 leader_election.cc:302] T 00000000000000000000000000000000 P 1c644b1536f34cbf9b7ce26eaef34710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c644b1536f34cbf9b7ce26eaef34710; no voters:
I20260925 22:41:05.709362 5331 leader_election.cc:288] T 00000000000000000000000000000000 P 1c644b1536f34cbf9b7ce26eaef34710 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:41:05.709484 5337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c644b1536f34cbf9b7ce26eaef34710 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:41:05.709769 5337 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c644b1536f34cbf9b7ce26eaef34710 [term 1 LEADER]: Becoming Leader. State: Replica: 1c644b1536f34cbf9b7ce26eaef34710, State: Running, Role: LEADER
I20260925 22:41:05.710153 5337 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c644b1536f34cbf9b7ce26eaef34710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c644b1536f34cbf9b7ce26eaef34710" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36331 } }
I20260925 22:41:05.710897 5331 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c644b1536f34cbf9b7ce26eaef34710 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:41:05.711397 5337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c644b1536f34cbf9b7ce26eaef34710 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c644b1536f34cbf9b7ce26eaef34710. Latest consensus state: current_term: 1 leader_uuid: "1c644b1536f34cbf9b7ce26eaef34710" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c644b1536f34cbf9b7ce26eaef34710" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36331 } } }
I20260925 22:41:05.711621 5337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c644b1536f34cbf9b7ce26eaef34710 [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:05.711861 5339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c644b1536f34cbf9b7ce26eaef34710 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c644b1536f34cbf9b7ce26eaef34710" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c644b1536f34cbf9b7ce26eaef34710" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36331 } } }
I20260925 22:41:05.712008 5339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c644b1536f34cbf9b7ce26eaef34710 [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:05.712085 5341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:41:05.715523 5341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:41:05.721778 5341 catalog_manager.cc:1347] Generated new cluster ID: 53054650087343789a4271c17dd15786
I20260925 22:41:05.721922 5341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:41:05.733072 5334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:05.723313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54956 (local address 127.2.74.254:36331)
0925 22:41:05.723420 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:05.723424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:05.727324 (+ 3900us) server_negotiation.cc:407] Connection header received
0925 22:41:05.727465 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:05.727469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:05.727527 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:05.727631 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:05.728269 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:05.729102 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:05.730428 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:05.730806 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:05.730912 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:05.731897 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:05.731901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:05.731904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:41:05.732532 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:05.732568 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:41:05.732588 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:05.732676 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:05.732845 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 22:41:05.732912 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 22:41:05.737664 5341 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:41:05.738660 5341 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:41:05.745224 5341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c644b1536f34cbf9b7ce26eaef34710: Generated new TSK 0
I20260925 22:41:05.746132 5341 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:41:05.757254 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 5263
2026-09-25T22:41:05Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1420 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:41:05 dist-test-slave-csnl krb5kdc[5364](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:41:05 dist-test-slave-csnl krb5kdc[5364](info): set up 2 sockets
Sep 25 22:41:05 dist-test-slave-csnl krb5kdc[5364](info): commencing operation
krb5kdc: starting...
W20260925 22:41:06.791167 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:41:06 dist-test-slave-csnl krb5kdc[5364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:41:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:41:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:41:06.918748 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:35497
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:45887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:35497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:07.033085 5377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:07.033435 5377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:07.033558 5377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:07.037887 5377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:41:07.037967 5377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:07.037988 5377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:07.038005 5377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:41:07.038021 5377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:41:07.043809 5377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:45887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:35497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:35497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:07.045008 5377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:07.046121 5377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:07.055482 5382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:07.055482 5383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:07.055500 5385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:07.055732 5377 server_base.cc:1034] running on GCE node
I20260925 22:41:07.056423 5377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:07.057731 5377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:07.058930 5377 hybrid_clock.cc:648] HybridClock initialized: now 1790376067058888 us; error 63 us; skew 500 ppm
Sep 25 22:41:07 dist-test-slave-csnl krb5kdc[5364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376067, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:07.062335 5377 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:41:07.063540 5377 webserver.cc:460] Webserver started at http://127.2.74.254:43191/ using document root <none> and password file <none>
I20260925 22:41:07.064204 5377 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:07.064317 5377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:07.064574 5377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:07.066881 5377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "97feb157fa564f4bb782a6c2c6f20435"
format_stamp: "Formatted at 2026-09-25 22:41:07 on dist-test-slave-csnl"
server_key: "8d701bae05a6c1ccc2e2f20990c66e08"
server_key_iv: "cd144309022c54596d974e546cf4f30f"
server_key_version: "encryptionkey@0"
I20260925 22:41:07.067615 5377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "97feb157fa564f4bb782a6c2c6f20435"
format_stamp: "Formatted at 2026-09-25 22:41:07 on dist-test-slave-csnl"
server_key: "8d701bae05a6c1ccc2e2f20990c66e08"
server_key_iv: "cd144309022c54596d974e546cf4f30f"
server_key_version: "encryptionkey@0"
I20260925 22:41:07.071848 5377 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:41:07.074748 5392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:07.076319 5377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:41:07.076599 5377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "97feb157fa564f4bb782a6c2c6f20435"
format_stamp: "Formatted at 2026-09-25 22:41:07 on dist-test-slave-csnl"
server_key: "8d701bae05a6c1ccc2e2f20990c66e08"
server_key_iv: "cd144309022c54596d974e546cf4f30f"
server_key_version: "encryptionkey@0"
I20260925 22:41:07.076757 5377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:07.106469 5377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:07.110049 5377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:07.110282 5377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:07.117873 5377 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:35497
I20260925 22:41:07.118218 5444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:35497 every 8 connection(s)
I20260925 22:41:07.118911 5377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:41:07.122005 5445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:07.126195 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 5377
I20260925 22:41:07.126327 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:41:07.126631 2347 external_mini_cluster.cc:1428] Setting key a75a31842f8cebe6e8c8d823baec4422
I20260925 22:41:07.128724 5445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97feb157fa564f4bb782a6c2c6f20435: Bootstrap starting.
I20260925 22:41:07.131028 5445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97feb157fa564f4bb782a6c2c6f20435: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:07.131786 5445 log.cc:824] T 00000000000000000000000000000000 P 97feb157fa564f4bb782a6c2c6f20435: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:07.134196 5445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97feb157fa564f4bb782a6c2c6f20435: No bootstrap required, opened a new log
Sep 25 22:41:07 dist-test-slave-csnl krb5kdc[5364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:41:07.138305 5445 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97feb157fa564f4bb782a6c2c6f20435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97feb157fa564f4bb782a6c2c6f20435" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 35497 } }
I20260925 22:41:07.138581 5445 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97feb157fa564f4bb782a6c2c6f20435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:07.138671 5445 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97feb157fa564f4bb782a6c2c6f20435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97feb157fa564f4bb782a6c2c6f20435, State: Initialized, Role: FOLLOWER
I20260925 22:41:07.139220 5445 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97feb157fa564f4bb782a6c2c6f20435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97feb157fa564f4bb782a6c2c6f20435" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 35497 } }
I20260925 22:41:07.139379 5445 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97feb157fa564f4bb782a6c2c6f20435 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:41:07.139451 5445 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97feb157fa564f4bb782a6c2c6f20435 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:41:07.139550 5445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97feb157fa564f4bb782a6c2c6f20435 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:07.140528 5445 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97feb157fa564f4bb782a6c2c6f20435 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97feb157fa564f4bb782a6c2c6f20435" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 35497 } }
I20260925 22:41:07.140951 5445 leader_election.cc:302] T 00000000000000000000000000000000 P 97feb157fa564f4bb782a6c2c6f20435 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97feb157fa564f4bb782a6c2c6f20435; no voters:
I20260925 22:41:07.141325 5445 leader_election.cc:288] T 00000000000000000000000000000000 P 97feb157fa564f4bb782a6c2c6f20435 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:41:07.141435 5450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97feb157fa564f4bb782a6c2c6f20435 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:41:07.141654 5450 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97feb157fa564f4bb782a6c2c6f20435 [term 1 LEADER]: Becoming Leader. State: Replica: 97feb157fa564f4bb782a6c2c6f20435, State: Running, Role: LEADER
I20260925 22:41:07.142017 5450 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97feb157fa564f4bb782a6c2c6f20435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97feb157fa564f4bb782a6c2c6f20435" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 35497 } }
I20260925 22:41:07.142181 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:07.128150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38444 (local address 127.2.74.254:35497)
0925 22:41:07.128622 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:07.128637 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:07.128672 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:41:07.129378 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:07.129397 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:07.129752 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:07.130114 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:07.130999 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:07.132106 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:07.133253 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:07.133866 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:07.134029 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:07.136837 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:07.136843 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:07.136860 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:07.136893 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:07.138799 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:07.139306 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:07.139312 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:07.139318 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:07.139490 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:07.139798 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:07.139803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:07.139804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:07.140222 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:07.140427 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:07.140737 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 22:41:07.141036 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":148,"threads_started":1}
I20260925 22:41:07.142688 5445 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97feb157fa564f4bb782a6c2c6f20435 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:41:07.143311 5452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97feb157fa564f4bb782a6c2c6f20435 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97feb157fa564f4bb782a6c2c6f20435. Latest consensus state: current_term: 1 leader_uuid: "97feb157fa564f4bb782a6c2c6f20435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97feb157fa564f4bb782a6c2c6f20435" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 35497 } } }
I20260925 22:41:07.143496 5452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97feb157fa564f4bb782a6c2c6f20435 [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:07.143776 5451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97feb157fa564f4bb782a6c2c6f20435 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97feb157fa564f4bb782a6c2c6f20435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97feb157fa564f4bb782a6c2c6f20435" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 35497 } } }
I20260925 22:41:07.143874 5451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97feb157fa564f4bb782a6c2c6f20435 [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:07.144701 5456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:41:07.149057 5456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:41:07.155390 5456 catalog_manager.cc:1347] Generated new cluster ID: 102be2c5232d47678021306f77457e2a
I20260925 22:41:07.155486 5456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:41:07.166127 5456 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:41:07.167550 5456 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:41:07.174683 5456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97feb157fa564f4bb782a6c2c6f20435: Generated new TSK 0
I20260925 22:41:07.175699 5456 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
I20260925 22:41:07.210668 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:0
--local_ip_for_outbound_sockets=127.2.74.193
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:35497
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:45887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:07.330291 5471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:07.330582 5471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:07.330627 5471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:07.334966 5471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:07.335044 5471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:07.335150 5471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:41:07.340893 5471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:45887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:35497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:07.342113 5471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:07.343366 5471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:07.354010 5479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:07.353986 5477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:07.353986 5476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:07.354051 5471 server_base.cc:1034] running on GCE node
I20260925 22:41:07.354802 5471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:07.355577 5471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:07.356787 5471 hybrid_clock.cc:648] HybridClock initialized: now 1790376067356768 us; error 36 us; skew 500 ppm
Sep 25 22:41:07 dist-test-slave-csnl krb5kdc[5364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376067, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:07.363648 5471 init.cc:377] Logged in from keytab as kudu/127.2.74.193@KRBTEST.COM (short username kudu)
I20260925 22:41:07.364781 5471 webserver.cc:460] Webserver started at http://127.2.74.193:42269/ using document root <none> and password file <none>
I20260925 22:41:07.365383 5471 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:07.365455 5471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:07.365641 5471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:07.368404 5471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/data/instance:
uuid: "3eb9e9ce13ea43a2aac14e2120d9d9e2"
format_stamp: "Formatted at 2026-09-25 22:41:07 on dist-test-slave-csnl"
server_key: "d230d87d67ea5dfa8ef72a9965035291"
server_key_iv: "12564119a7a8447423499d4bd3293788"
server_key_version: "encryptionkey@0"
I20260925 22:41:07.369029 5471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance:
uuid: "3eb9e9ce13ea43a2aac14e2120d9d9e2"
format_stamp: "Formatted at 2026-09-25 22:41:07 on dist-test-slave-csnl"
server_key: "d230d87d67ea5dfa8ef72a9965035291"
server_key_iv: "12564119a7a8447423499d4bd3293788"
server_key_version: "encryptionkey@0"
I20260925 22:41:07.373960 5471 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 22:41:07.378152 5486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:07.380178 5471 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:41:07.380373 5471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "3eb9e9ce13ea43a2aac14e2120d9d9e2"
format_stamp: "Formatted at 2026-09-25 22:41:07 on dist-test-slave-csnl"
server_key: "d230d87d67ea5dfa8ef72a9965035291"
server_key_iv: "12564119a7a8447423499d4bd3293788"
server_key_version: "encryptionkey@0"
I20260925 22:41:07.380540 5471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:07.398583 5471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:07.402133 5471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:07.402369 5471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:07.403065 5471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:07.403939 5471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:07.403988 5471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:07.404057 5471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:07.404088 5471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:07.413520 5471 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:39547
I20260925 22:41:07.413537 5599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:39547 every 8 connection(s)
I20260925 22:41:07.414564 5471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:41:07.418056 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 5471
I20260925 22:41:07.418196 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance
I20260925 22:41:07.418552 2347 external_mini_cluster.cc:1428] Setting key f81af2574dc077d0a4dd00b34f2978bb
Sep 25 22:41:07 dist-test-slave-csnl krb5kdc[5364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376067, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.74.194@KRBTEST.COM; defaulting to no policy
I20260925 22:41:07.431627 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:07.416574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:55395 (local address 127.2.74.254:35497)
0925 22:41:07.416728 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:07.416732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:07.417695 (+ 963us) server_negotiation.cc:407] Connection header received
0925 22:41:07.419601 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:07.419604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:07.419653 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:07.419730 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:07.421763 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:07.422532 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:07.423939 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:07.424183 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:07.424327 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:07.427482 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:07.427487 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:07.427491 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:07.427533 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:07.429275 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:07.429980 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:07.429983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:07.429984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:07.430065 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:07.430397 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:07.430400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:07.430402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:07.430598 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:07.430675 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:07.431280 (+ 605us) server_negotiation.cc:299] Negotiation successful
0925 22:41:07.431438 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.194@KRBTEST.COM" created.
I20260925 22:41:07.432578 5602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:07.417039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:35497 (local address 127.2.74.193:55395)
0925 22:41:07.417535 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:07.417583 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:07.419321 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:07.419973 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:07.419986 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:07.420502 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:07.421309 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:07.421334 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:07.422698 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:07.422701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:07.423776 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:07.423788 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:07.424396 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:07.424400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:07.424584 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:07.425356 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:07.425374 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:07.427243 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:07.429537 (+ 2294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:07.429549 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:07.429566 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:07.429861 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:07.430149 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:07.430151 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:07.430152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:07.430298 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:07.430690 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:07.430695 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:07.431036 (+ 341us) client_negotiation.cc:769] Sending connection context
0925 22:41:07.431209 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 22:41:07.431562 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":125,"threads_started":1}
I20260925 22:41:07.433923 5600 heartbeater.cc:347] Connected to a master server at 127.2.74.254:35497
I20260925 22:41:07.434196 5600 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:07.434789 5600 heartbeater.cc:511] Master 127.2.74.254:35497 requested a full tablet report, sending...
I20260925 22:41:07.436621 5409 ts_manager.cc:194] Registered new tserver with Master: 3eb9e9ce13ea43a2aac14e2120d9d9e2 (127.2.74.193:39547)
I20260925 22:41:07.438390 5409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.193@KRBTEST.COM'} at 127.2.74.193:55395
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
I20260925 22:41:07.450652 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:0
--local_ip_for_outbound_sockets=127.2.74.194
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:35497
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:45887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:07.564913 5605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:07.565215 5605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:07.565261 5605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:07.569326 5605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:07.569401 5605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:07.569464 5605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:41:07.575045 5605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:45887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:35497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:07.576141 5605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:07.577255 5605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:07.587773 5610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:07.587837 5611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:07.588174 5605 server_base.cc:1034] running on GCE node
W20260925 22:41:07.587773 5613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:07.588697 5605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:07.589416 5605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:07.590615 5605 hybrid_clock.cc:648] HybridClock initialized: now 1790376067590595 us; error 37 us; skew 500 ppm
Sep 25 22:41:07 dist-test-slave-csnl krb5kdc[5364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376067, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:07.593737 5605 init.cc:377] Logged in from keytab as kudu/127.2.74.194@KRBTEST.COM (short username kudu)
I20260925 22:41:07.594784 5605 webserver.cc:460] Webserver started at http://127.2.74.194:33681/ using document root <none> and password file <none>
I20260925 22:41:07.595482 5605 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:07.595587 5605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:07.595840 5605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:07.598156 5605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/data/instance:
uuid: "96b6bd18540a476cbf96fdfa7fb1cfaf"
format_stamp: "Formatted at 2026-09-25 22:41:07 on dist-test-slave-csnl"
server_key: "036397c218295b851c8bc981b94905ab"
server_key_iv: "f6517dd6e4fbf6280b2d2c603e1ce8c2"
server_key_version: "encryptionkey@0"
I20260925 22:41:07.598807 5605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance:
uuid: "96b6bd18540a476cbf96fdfa7fb1cfaf"
format_stamp: "Formatted at 2026-09-25 22:41:07 on dist-test-slave-csnl"
server_key: "036397c218295b851c8bc981b94905ab"
server_key_iv: "f6517dd6e4fbf6280b2d2c603e1ce8c2"
server_key_version: "encryptionkey@0"
I20260925 22:41:07.603071 5605 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:41:07.605793 5620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:07.607311 5605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:41:07.607478 5605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "96b6bd18540a476cbf96fdfa7fb1cfaf"
format_stamp: "Formatted at 2026-09-25 22:41:07 on dist-test-slave-csnl"
server_key: "036397c218295b851c8bc981b94905ab"
server_key_iv: "f6517dd6e4fbf6280b2d2c603e1ce8c2"
server_key_version: "encryptionkey@0"
I20260925 22:41:07.607614 5605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:07.621553 5605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:07.624928 5605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:07.625161 5605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:07.625815 5605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:07.626729 5605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:07.626806 5605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:07.626880 5605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:07.626940 5605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:07.636436 5605 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:41677
I20260925 22:41:07.636457 5733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:41677 every 8 connection(s)
I20260925 22:41:07.637504 5605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:41:07.647467 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 5605
I20260925 22:41:07.647615 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance
Sep 25 22:41:07 dist-test-slave-csnl krb5kdc[5364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376067, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:41:07.647969 2347 external_mini_cluster.cc:1428] Setting key 2949bde8320371af36a1e3ab93632f81
I20260925 22:41:07.653061 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:07.639427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:46481 (local address 127.2.74.254:35497)
0925 22:41:07.639594 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:07.639599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:07.640538 (+ 939us) server_negotiation.cc:407] Connection header received
0925 22:41:07.641921 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:07.641924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:07.641975 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:07.642052 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:07.643448 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:07.644038 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:07.645323 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:07.645539 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:07.645624 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:07.648542 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:07.648549 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:07.648553 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:07.648592 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:07.650163 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:07.650910 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:07.650912 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:07.650913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:07.651062 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:07.651536 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:07.651541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:07.651543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:07.651839 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:07.651956 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:07.652628 (+ 672us) server_negotiation.cc:299] Negotiation successful
0925 22:41:07.652803 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 22:41:07.653812 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:07.639881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:35497 (local address 127.2.74.194:46481)
0925 22:41:07.640375 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:07.640421 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:07.641706 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:07.642226 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:07.642233 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:07.642627 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:07.643276 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:07.643293 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:07.644154 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:07.644157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:07.645215 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:07.645225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:07.645678 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:07.645680 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:07.645830 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:07.646453 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:07.646471 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:07.648281 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:07.650307 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:07.650317 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:07.650352 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:07.650793 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:07.651156 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:07.651163 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:07.651168 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:07.651418 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:07.651953 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:07.651959 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:07.652348 (+ 389us) client_negotiation.cc:769] Sending connection context
0925 22:41:07.652534 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 22:41:07.652829 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":124,"threads_started":1}
I20260925 22:41:07.655171 5734 heartbeater.cc:347] Connected to a master server at 127.2.74.254:35497
I20260925 22:41:07.655510 5734 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:07.656262 5734 heartbeater.cc:511] Master 127.2.74.254:35497 requested a full tablet report, sending...
I20260925 22:41:07.657665 5409 ts_manager.cc:194] Registered new tserver with Master: 96b6bd18540a476cbf96fdfa7fb1cfaf (127.2.74.194:41677)
I20260925 22:41:07.658329 5409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.194@KRBTEST.COM'} at 127.2.74.194:46481
WARNING: no policy specified for kudu/127.2.74.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
I20260925 22:41:07.678364 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:0
--local_ip_for_outbound_sockets=127.2.74.195
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:35497
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:45887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:07.795626 5739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:07.795935 5739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:07.795982 5739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:07.800149 5739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:07.800225 5739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:07.800287 5739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:41:07.806083 5739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:45887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:35497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:07.807225 5739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:07.808235 5739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:41:07.818768 5739 server_base.cc:1034] running on GCE node
W20260925 22:41:07.818701 5747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:07.818730 5745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:07.818702 5744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:07.819453 5739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:07.820122 5739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:07.821290 5739 hybrid_clock.cc:648] HybridClock initialized: now 1790376067821272 us; error 35 us; skew 500 ppm
Sep 25 22:41:07 dist-test-slave-csnl krb5kdc[5364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376067, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:07.829631 5739 init.cc:377] Logged in from keytab as kudu/127.2.74.195@KRBTEST.COM (short username kudu)
I20260925 22:41:07.830684 5739 webserver.cc:460] Webserver started at http://127.2.74.195:37571/ using document root <none> and password file <none>
I20260925 22:41:07.831363 5739 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:07.831434 5739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:07.831697 5739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:07.834003 5739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/data/instance:
uuid: "88701f0bbb2d40d1be32fc7c56862e9a"
format_stamp: "Formatted at 2026-09-25 22:41:07 on dist-test-slave-csnl"
server_key: "d0e85fe9871803878ba73fe88a49d64d"
server_key_iv: "d6de4afe72663992a9cf7814db5918ef"
server_key_version: "encryptionkey@0"
I20260925 22:41:07.834614 5739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance:
uuid: "88701f0bbb2d40d1be32fc7c56862e9a"
format_stamp: "Formatted at 2026-09-25 22:41:07 on dist-test-slave-csnl"
server_key: "d0e85fe9871803878ba73fe88a49d64d"
server_key_iv: "d6de4afe72663992a9cf7814db5918ef"
server_key_version: "encryptionkey@0"
I20260925 22:41:07.838595 5739 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 22:41:07.841255 5754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:07.842775 5739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:41:07.842918 5739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "88701f0bbb2d40d1be32fc7c56862e9a"
format_stamp: "Formatted at 2026-09-25 22:41:07 on dist-test-slave-csnl"
server_key: "d0e85fe9871803878ba73fe88a49d64d"
server_key_iv: "d6de4afe72663992a9cf7814db5918ef"
server_key_version: "encryptionkey@0"
I20260925 22:41:07.843111 5739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:07.870954 5739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:07.875238 5739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:07.875444 5739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:07.876122 5739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:07.877122 5739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:07.877182 5739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:07.877434 5739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:07.877473 5739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:07.890115 5739 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:43933
I20260925 22:41:07.890168 5867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:43933 every 8 connection(s)
I20260925 22:41:07.891335 5739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:41:07.895787 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 5739
I20260925 22:41:07.895931 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance
I20260925 22:41:07.896246 2347 external_mini_cluster.cc:1428] Setting key fac275c3ad3229ada18d15c2a063fc67
Sep 25 22:41:07 dist-test-slave-csnl krb5kdc[5364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376067, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:41:07.910267 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:07.893601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:49637 (local address 127.2.74.254:35497)
0925 22:41:07.893747 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:07.893752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:07.895144 (+ 1392us) server_negotiation.cc:407] Connection header received
0925 22:41:07.897271 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:07.897274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:07.897337 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:07.897437 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:07.899152 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:07.900150 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:07.901448 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:07.901864 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:07.901970 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:07.905872 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:07.905875 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:07.905878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:07.905942 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:07.907785 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:07.908385 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:07.908390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:07.908392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:07.908487 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:07.908887 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:07.908890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:07.908892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:07.909097 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:07.909213 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:07.909860 (+ 647us) server_negotiation.cc:299] Negotiation successful
0925 22:41:07.910033 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:41:07.911394 5870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:07.894159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:35497 (local address 127.2.74.195:49637)
0925 22:41:07.894904 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:07.895014 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:07.897002 (+ 1988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:07.897601 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:07.897609 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:07.898213 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:07.898950 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:07.899010 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:07.900311 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:07.900315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:07.901295 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:07.901305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:07.902046 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:07.902050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:07.902258 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:07.903022 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:07.903044 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:07.905604 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:07.907921 (+ 2317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:07.907928 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:07.907948 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:07.908254 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:07.908582 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:07.908586 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:07.908588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:07.908756 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:07.909224 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:07.909233 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:07.909526 (+ 293us) client_negotiation.cc:769] Sending connection context
0925 22:41:07.909743 (+ 217us) client_negotiation.cc:240] Negotiation successful
0925 22:41:07.910135 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":170,"threads_started":1}
I20260925 22:41:07.912817 5868 heartbeater.cc:347] Connected to a master server at 127.2.74.254:35497
I20260925 22:41:07.913136 5868 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:07.913882 5868 heartbeater.cc:511] Master 127.2.74.254:35497 requested a full tablet report, sending...
I20260925 22:41:07.915427 5409 ts_manager.cc:194] Registered new tserver with Master: 88701f0bbb2d40d1be32fc7c56862e9a (127.2.74.195:43933)
I20260925 22:41:07.916245 5409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.195@KRBTEST.COM'} at 127.2.74.195:49637
I20260925 22:41:07.923200 2347 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 22:41:07 dist-test-slave-csnl krb5kdc[5364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376067, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 22:41:07 dist-test-slave-csnl krb5kdc[5364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376067, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:41:07.949910 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:07.940983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50736 (local address 127.2.74.254:35497)
0925 22:41:07.941116 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:07.941120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:07.941268 (+ 148us) server_negotiation.cc:407] Connection header received
0925 22:41:07.941398 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:07.941400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:07.941446 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:07.941525 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:07.942203 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:07.942764 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:07.943697 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:07.943955 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:07.944079 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:07.946623 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:07.946626 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:07.946628 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:07.946657 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:07.948329 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:07.948770 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:07.948773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:07.948774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:07.948861 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:07.949095 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:07.949097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:07.949099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:07.949323 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:07.949436 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:07.949664 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 22:41:07.949777 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 22:41:07.950367 5409 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50736
I20260925 22:41:07.951313 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 5471
I20260925 22:41:07.956799 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 5605
I20260925 22:41:07.962065 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 5739
I20260925 22:41:07.967567 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 5377
2026-09-25T22:41:07Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2211 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:41:08 dist-test-slave-csnl krb5kdc[5880](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:41:08 dist-test-slave-csnl krb5kdc[5880](info): set up 2 sockets
Sep 25 22:41:08 dist-test-slave-csnl krb5kdc[5880](info): commencing operation
krb5kdc: starting...
W20260925 22:41:09.007170 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:41:09 dist-test-slave-csnl krb5kdc[5880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:41:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:41:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:41:09.133625 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:46355
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:39227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:46355
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:09.246905 5893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:09.247304 5893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:09.247349 5893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:09.251552 5893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:41:09.251617 5893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:09.251637 5893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:09.251657 5893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:41:09.251672 5893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:41:09.257166 5893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:39227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:46355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:46355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:09.258250 5893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:09.259356 5893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:09.268500 5899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:09.268507 5898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:09.268599 5893 server_base.cc:1034] running on GCE node
W20260925 22:41:09.268500 5901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:09.269412 5893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:09.270597 5893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:09.271773 5893 hybrid_clock.cc:648] HybridClock initialized: now 1790376069271755 us; error 34 us; skew 500 ppm
Sep 25 22:41:09 dist-test-slave-csnl krb5kdc[5880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376069, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:09.274776 5893 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:41:09.275867 5893 webserver.cc:460] Webserver started at http://127.2.74.254:44107/ using document root <none> and password file <none>
I20260925 22:41:09.276502 5893 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:09.276623 5893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:09.276885 5893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:09.279191 5893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "c70990d648584c33848b7e5ef28a7de9"
format_stamp: "Formatted at 2026-09-25 22:41:09 on dist-test-slave-csnl"
server_key: "240b2a579a70775327e9f33367f21547"
server_key_iv: "aa7029c13a2c307cf608717df462d089"
server_key_version: "encryptionkey@0"
I20260925 22:41:09.279814 5893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "c70990d648584c33848b7e5ef28a7de9"
format_stamp: "Formatted at 2026-09-25 22:41:09 on dist-test-slave-csnl"
server_key: "240b2a579a70775327e9f33367f21547"
server_key_iv: "aa7029c13a2c307cf608717df462d089"
server_key_version: "encryptionkey@0"
I20260925 22:41:09.284135 5893 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:41:09.287125 5908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:09.288582 5893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:41:09.288762 5893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "c70990d648584c33848b7e5ef28a7de9"
format_stamp: "Formatted at 2026-09-25 22:41:09 on dist-test-slave-csnl"
server_key: "240b2a579a70775327e9f33367f21547"
server_key_iv: "aa7029c13a2c307cf608717df462d089"
server_key_version: "encryptionkey@0"
I20260925 22:41:09.288908 5893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:09.309615 5893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:09.313333 5893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:09.313599 5893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:09.322327 5893 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:46355
I20260925 22:41:09.322376 5960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:46355 every 8 connection(s)
I20260925 22:41:09.323581 5893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:41:09.326592 5961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:09.330341 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 5893
I20260925 22:41:09.330473 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:41:09.330739 2347 external_mini_cluster.cc:1428] Setting key 0e21007db05a5d790dc3d9194dd83f6d
I20260925 22:41:09.333094 5961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c70990d648584c33848b7e5ef28a7de9: Bootstrap starting.
I20260925 22:41:09.335652 5961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c70990d648584c33848b7e5ef28a7de9: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:09.336490 5961 log.cc:824] T 00000000000000000000000000000000 P c70990d648584c33848b7e5ef28a7de9: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:09.338658 5961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c70990d648584c33848b7e5ef28a7de9: No bootstrap required, opened a new log
Sep 25 22:41:09 dist-test-slave-csnl krb5kdc[5880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:41:09.342803 5961 raft_consensus.cc:348] T 00000000000000000000000000000000 P c70990d648584c33848b7e5ef28a7de9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c70990d648584c33848b7e5ef28a7de9" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 46355 } }
I20260925 22:41:09.343163 5961 raft_consensus.cc:374] T 00000000000000000000000000000000 P c70990d648584c33848b7e5ef28a7de9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:09.343257 5961 raft_consensus.cc:729] T 00000000000000000000000000000000 P c70990d648584c33848b7e5ef28a7de9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c70990d648584c33848b7e5ef28a7de9, State: Initialized, Role: FOLLOWER
I20260925 22:41:09.343794 5961 consensus_queue.cc:260] T 00000000000000000000000000000000 P c70990d648584c33848b7e5ef28a7de9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c70990d648584c33848b7e5ef28a7de9" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 46355 } }
I20260925 22:41:09.343973 5961 raft_consensus.cc:388] T 00000000000000000000000000000000 P c70990d648584c33848b7e5ef28a7de9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:41:09.344049 5961 raft_consensus.cc:482] T 00000000000000000000000000000000 P c70990d648584c33848b7e5ef28a7de9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:41:09.344164 5961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c70990d648584c33848b7e5ef28a7de9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:09.345383 5961 raft_consensus.cc:504] T 00000000000000000000000000000000 P c70990d648584c33848b7e5ef28a7de9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c70990d648584c33848b7e5ef28a7de9" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 46355 } }
I20260925 22:41:09.345813 5961 leader_election.cc:302] T 00000000000000000000000000000000 P c70990d648584c33848b7e5ef28a7de9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c70990d648584c33848b7e5ef28a7de9; no voters:
I20260925 22:41:09.346232 5961 leader_election.cc:288] T 00000000000000000000000000000000 P c70990d648584c33848b7e5ef28a7de9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:41:09.346331 5966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c70990d648584c33848b7e5ef28a7de9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:41:09.346544 5966 raft_consensus.cc:686] T 00000000000000000000000000000000 P c70990d648584c33848b7e5ef28a7de9 [term 1 LEADER]: Becoming Leader. State: Replica: c70990d648584c33848b7e5ef28a7de9, State: Running, Role: LEADER
I20260925 22:41:09.346899 5966 consensus_queue.cc:237] T 00000000000000000000000000000000 P c70990d648584c33848b7e5ef28a7de9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c70990d648584c33848b7e5ef28a7de9" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 46355 } }
I20260925 22:41:09.347282 5964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:09.332357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43946 (local address 127.2.74.254:46355)
0925 22:41:09.332872 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:09.332888 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:09.332928 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:41:09.333708 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:09.333729 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:09.334112 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:09.334501 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:09.335418 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:09.336548 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:09.337480 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:09.337985 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:09.338174 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:09.341220 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:09.341227 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:09.341244 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:09.341279 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:09.343804 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:09.344392 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:09.344397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:09.344403 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:09.344526 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:09.344846 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:09.344848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:09.344850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:09.345272 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:09.345434 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:09.345760 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 22:41:09.346094 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":157,"threads_started":1}
I20260925 22:41:09.347739 5961 sys_catalog.cc:564] T 00000000000000000000000000000000 P c70990d648584c33848b7e5ef28a7de9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:41:09.348232 5966 sys_catalog.cc:455] T 00000000000000000000000000000000 P c70990d648584c33848b7e5ef28a7de9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c70990d648584c33848b7e5ef28a7de9. Latest consensus state: current_term: 1 leader_uuid: "c70990d648584c33848b7e5ef28a7de9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c70990d648584c33848b7e5ef28a7de9" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 46355 } } }
I20260925 22:41:09.348356 5966 sys_catalog.cc:458] T 00000000000000000000000000000000 P c70990d648584c33848b7e5ef28a7de9 [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:09.348982 5967 sys_catalog.cc:455] T 00000000000000000000000000000000 P c70990d648584c33848b7e5ef28a7de9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c70990d648584c33848b7e5ef28a7de9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c70990d648584c33848b7e5ef28a7de9" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 46355 } } }
I20260925 22:41:09.349085 5967 sys_catalog.cc:458] T 00000000000000000000000000000000 P c70990d648584c33848b7e5ef28a7de9 [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:09.349822 5970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:41:09.353293 5970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:41:09.359512 5970 catalog_manager.cc:1347] Generated new cluster ID: 35679182985146e38f762a7f5b99d357
I20260925 22:41:09.359606 5970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:41:09.367712 5970 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:41:09.369103 5970 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:41:09.378598 5970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c70990d648584c33848b7e5ef28a7de9: Generated new TSK 0
I20260925 22:41:09.379581 5970 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
I20260925 22:41:09.414145 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:0
--local_ip_for_outbound_sockets=127.2.74.193
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:46355
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:39227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:09.532840 5987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:09.533196 5987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:09.533285 5987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:09.537706 5987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:09.537833 5987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:09.537940 5987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:41:09.543889 5987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:39227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:46355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:09.545070 5987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:09.546164 5987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:09.556592 5993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:09.556592 5992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:09.556630 5995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:09.557047 5987 server_base.cc:1034] running on GCE node
I20260925 22:41:09.557472 5987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:09.558185 5987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:09.559357 5987 hybrid_clock.cc:648] HybridClock initialized: now 1790376069559333 us; error 45 us; skew 500 ppm
Sep 25 22:41:09 dist-test-slave-csnl krb5kdc[5880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376069, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:09.562337 5987 init.cc:377] Logged in from keytab as kudu/127.2.74.193@KRBTEST.COM (short username kudu)
I20260925 22:41:09.563401 5987 webserver.cc:460] Webserver started at http://127.2.74.193:42623/ using document root <none> and password file <none>
I20260925 22:41:09.563990 5987 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:09.564056 5987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:09.564229 5987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:09.566648 5987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-0/data/instance:
uuid: "f8d857d6559a4112aa0f43006c65974b"
format_stamp: "Formatted at 2026-09-25 22:41:09 on dist-test-slave-csnl"
server_key: "7b9bd6c7aae252ae79b2acab76b04f58"
server_key_iv: "423465d75e149a50f0eb15534a87f7d0"
server_key_version: "encryptionkey@0"
I20260925 22:41:09.567273 5987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance:
uuid: "f8d857d6559a4112aa0f43006c65974b"
format_stamp: "Formatted at 2026-09-25 22:41:09 on dist-test-slave-csnl"
server_key: "7b9bd6c7aae252ae79b2acab76b04f58"
server_key_iv: "423465d75e149a50f0eb15534a87f7d0"
server_key_version: "encryptionkey@0"
I20260925 22:41:09.571236 5987 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:41:09.573849 6002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:09.575313 5987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:41:09.575457 5987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "f8d857d6559a4112aa0f43006c65974b"
format_stamp: "Formatted at 2026-09-25 22:41:09 on dist-test-slave-csnl"
server_key: "7b9bd6c7aae252ae79b2acab76b04f58"
server_key_iv: "423465d75e149a50f0eb15534a87f7d0"
server_key_version: "encryptionkey@0"
I20260925 22:41:09.575588 5987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:09.594601 5987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:09.598408 5987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:09.598649 5987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:09.599350 5987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:09.600253 5987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:09.600302 5987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:09.600373 5987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:09.600404 5987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:09.610347 5987 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:41121
I20260925 22:41:09.610373 6115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:41121 every 8 connection(s)
I20260925 22:41:09.611375 5987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:41:09.611572 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 5987
I20260925 22:41:09.611728 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance
I20260925 22:41:09.612097 2347 external_mini_cluster.cc:1428] Setting key 51b1fced80c87884539886815c9a6572
WARNING: no policy specified for kudu/127.2.74.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.194@KRBTEST.COM" created.
Sep 25 22:41:09 dist-test-slave-csnl krb5kdc[5880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376069, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:41:09.631134 5964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:09.613549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:53237 (local address 127.2.74.254:46355)
0925 22:41:09.613683 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:09.613688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:09.614854 (+ 1166us) server_negotiation.cc:407] Connection header received
0925 22:41:09.616632 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:09.616635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:09.616684 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:09.616769 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:09.618662 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:09.619264 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:09.621343 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:09.621569 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:09.621652 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:09.626884 (+ 5232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:09.626891 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:09.626898 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:09.626938 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:09.628590 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:09.629190 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:09.629197 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:09.629202 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:09.629308 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:09.629673 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:09.629679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:09.629683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:09.629983 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:09.630103 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:09.630649 (+ 546us) server_negotiation.cc:299] Negotiation successful
0925 22:41:09.630823 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:41:09.631942 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:09.614096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:46355 (local address 127.2.74.193:53237)
0925 22:41:09.614636 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:09.614700 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:09.616372 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:09.616983 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:09.616996 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:09.617624 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:09.618473 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:09.618500 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:09.619413 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:09.619418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:09.621189 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:09.621207 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:09.621749 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:09.621754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:09.622037 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:09.622876 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:09.622901 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:09.626643 (+ 3742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:09.628749 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:09.628754 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:09.628770 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:09.629067 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:09.629414 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:09.629416 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:09.629418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:09.629569 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:09.630094 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:09.630100 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:09.630386 (+ 286us) client_negotiation.cc:769] Sending connection context
0925 22:41:09.630564 (+ 178us) client_negotiation.cc:240] Negotiation successful
0925 22:41:09.630856 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":162,"threads_started":1}
I20260925 22:41:09.633386 6116 heartbeater.cc:347] Connected to a master server at 127.2.74.254:46355
I20260925 22:41:09.633659 6116 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:09.634193 6116 heartbeater.cc:511] Master 127.2.74.254:46355 requested a full tablet report, sending...
I20260925 22:41:09.636003 5925 ts_manager.cc:194] Registered new tserver with Master: f8d857d6559a4112aa0f43006c65974b (127.2.74.193:41121)
I20260925 22:41:09.637411 5925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.193@KRBTEST.COM'} at 127.2.74.193:53237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
I20260925 22:41:09.643404 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:0
--local_ip_for_outbound_sockets=127.2.74.194
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:46355
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:39227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:09.757555 6121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:09.757853 6121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:09.757938 6121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:09.762189 6121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:09.762270 6121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:09.762398 6121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:41:09.767938 6121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:39227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:46355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:09.769099 6121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:09.770180 6121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:09.780231 6126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:09.780212 6129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:09.780190 6127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:09.780412 6121 server_base.cc:1034] running on GCE node
I20260925 22:41:09.781047 6121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:09.781687 6121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:09.782845 6121 hybrid_clock.cc:648] HybridClock initialized: now 1790376069782822 us; error 37 us; skew 500 ppm
Sep 25 22:41:09 dist-test-slave-csnl krb5kdc[5880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376069, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:09.785759 6121 init.cc:377] Logged in from keytab as kudu/127.2.74.194@KRBTEST.COM (short username kudu)
I20260925 22:41:09.786829 6121 webserver.cc:460] Webserver started at http://127.2.74.194:41683/ using document root <none> and password file <none>
I20260925 22:41:09.787503 6121 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:09.787576 6121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:09.787765 6121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:09.790112 6121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-1/data/instance:
uuid: "ae812549d1c64da5a9b27d0ad62408c4"
format_stamp: "Formatted at 2026-09-25 22:41:09 on dist-test-slave-csnl"
server_key: "54e07e69334ac76e7c1f6c82c85e378e"
server_key_iv: "d8b0f23719d1f69346604c0ffa8aa714"
server_key_version: "encryptionkey@0"
I20260925 22:41:09.790725 6121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance:
uuid: "ae812549d1c64da5a9b27d0ad62408c4"
format_stamp: "Formatted at 2026-09-25 22:41:09 on dist-test-slave-csnl"
server_key: "54e07e69334ac76e7c1f6c82c85e378e"
server_key_iv: "d8b0f23719d1f69346604c0ffa8aa714"
server_key_version: "encryptionkey@0"
I20260925 22:41:09.795405 6121 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:41:09.798053 6136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:09.799496 6121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:41:09.799649 6121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "ae812549d1c64da5a9b27d0ad62408c4"
format_stamp: "Formatted at 2026-09-25 22:41:09 on dist-test-slave-csnl"
server_key: "54e07e69334ac76e7c1f6c82c85e378e"
server_key_iv: "d8b0f23719d1f69346604c0ffa8aa714"
server_key_version: "encryptionkey@0"
I20260925 22:41:09.799773 6121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:09.811836 6121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:09.815538 6121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:09.815773 6121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:09.816444 6121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:09.817348 6121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:09.817404 6121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:09.817473 6121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:09.817507 6121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:09.826716 6121 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:44511
I20260925 22:41:09.826761 6249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:44511 every 8 connection(s)
I20260925 22:41:09.827739 6121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:41:09.829988 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 6121
I20260925 22:41:09.830135 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance
I20260925 22:41:09.830432 2347 external_mini_cluster.cc:1428] Setting key 7eca54431960ed44563546a8e2741da4
Sep 25 22:41:09 dist-test-slave-csnl krb5kdc[5880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376069, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.74.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.195@KRBTEST.COM" created.
I20260925 22:41:09.843699 5964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:09.829722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:52933 (local address 127.2.74.254:46355)
0925 22:41:09.829911 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:09.829917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:09.831118 (+ 1201us) server_negotiation.cc:407] Connection header received
0925 22:41:09.832941 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:09.832944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:09.832989 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:09.833063 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:09.834552 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:09.835158 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:09.836476 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:09.836709 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:09.836829 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:09.839721 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:09.839725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:09.839728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:09.839763 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:09.841315 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:09.841957 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:09.841960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:09.841961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:09.842038 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:09.842428 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:09.842432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:09.842434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:09.842638 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:09.842754 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:09.843384 (+ 630us) server_negotiation.cc:299] Negotiation successful
0925 22:41:09.843526 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 22:41:09.844674 6252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:09.830294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:46355 (local address 127.2.74.194:52933)
0925 22:41:09.830878 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:09.830939 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:09.832676 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:09.833256 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:09.833268 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:09.833839 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:09.834396 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:09.834410 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:09.835297 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:09.835300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:09.836337 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:09.836348 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:09.836879 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:09.836883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:09.837071 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:09.837723 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:09.837741 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:09.839490 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:09.841444 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:09.841449 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:09.841466 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:09.841861 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:09.842142 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:09.842148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:09.842151 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:09.842337 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:09.842764 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:09.842773 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:09.843120 (+ 347us) client_negotiation.cc:769] Sending connection context
0925 22:41:09.843327 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 22:41:09.843694 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":156,"threads_started":1}
I20260925 22:41:09.845991 6250 heartbeater.cc:347] Connected to a master server at 127.2.74.254:46355
I20260925 22:41:09.846293 6250 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:09.847047 6250 heartbeater.cc:511] Master 127.2.74.254:46355 requested a full tablet report, sending...
I20260925 22:41:09.848330 5925 ts_manager.cc:194] Registered new tserver with Master: ae812549d1c64da5a9b27d0ad62408c4 (127.2.74.194:44511)
I20260925 22:41:09.849305 5925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.194@KRBTEST.COM'} at 127.2.74.194:52933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
I20260925 22:41:09.860095 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:0
--local_ip_for_outbound_sockets=127.2.74.195
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:46355
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:39227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:09.973227 6255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:09.973537 6255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:09.973582 6255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:09.977886 6255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:09.977967 6255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:09.978029 6255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:41:09.984196 6255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:39227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:46355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:09.985342 6255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:09.986465 6255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:09.997303 6261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:09.997404 6255 server_base.cc:1034] running on GCE node
W20260925 22:41:09.997337 6260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:09.997324 6263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:09.997980 6255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:09.998728 6255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:09.999909 6255 hybrid_clock.cc:648] HybridClock initialized: now 1790376069999892 us; error 43 us; skew 500 ppm
Sep 25 22:41:09 dist-test-slave-csnl krb5kdc[5880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376069, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:10.002827 6255 init.cc:377] Logged in from keytab as kudu/127.2.74.195@KRBTEST.COM (short username kudu)
I20260925 22:41:10.003984 6255 webserver.cc:460] Webserver started at http://127.2.74.195:40327/ using document root <none> and password file <none>
I20260925 22:41:10.004685 6255 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:10.004783 6255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:10.005033 6255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:10.007344 6255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-2/data/instance:
uuid: "aa5ad0e00da9446e8d82ba2f025a412a"
format_stamp: "Formatted at 2026-09-25 22:41:10 on dist-test-slave-csnl"
server_key: "0549c5a6f9077a2023175d6b7075b021"
server_key_iv: "336392a82e73748df9971d71929b20e4"
server_key_version: "encryptionkey@0"
I20260925 22:41:10.008028 6255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance:
uuid: "aa5ad0e00da9446e8d82ba2f025a412a"
format_stamp: "Formatted at 2026-09-25 22:41:10 on dist-test-slave-csnl"
server_key: "0549c5a6f9077a2023175d6b7075b021"
server_key_iv: "336392a82e73748df9971d71929b20e4"
server_key_version: "encryptionkey@0"
I20260925 22:41:10.012262 6255 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:41:10.015180 6270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:10.016561 6255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:41:10.016752 6255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "aa5ad0e00da9446e8d82ba2f025a412a"
format_stamp: "Formatted at 2026-09-25 22:41:10 on dist-test-slave-csnl"
server_key: "0549c5a6f9077a2023175d6b7075b021"
server_key_iv: "336392a82e73748df9971d71929b20e4"
server_key_version: "encryptionkey@0"
I20260925 22:41:10.016893 6255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:10.039686 6255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:10.043102 6255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:10.043371 6255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:10.044027 6255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:10.044945 6255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:10.045023 6255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:10.045096 6255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:10.045140 6255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:10.055161 6255 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:41477
I20260925 22:41:10.055219 6383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:41477 every 8 connection(s)
I20260925 22:41:10.056209 6255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:41:10.056933 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 6255
I20260925 22:41:10.057070 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance
I20260925 22:41:10.057381 2347 external_mini_cluster.cc:1428] Setting key 2f63ef8cd32d500a093d77415a5f9a0b
Sep 25 22:41:10 dist-test-slave-csnl krb5kdc[5880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376069, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:41:10.072676 5964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:10.059177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:60025 (local address 127.2.74.254:46355)
0925 22:41:10.060145 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:10.060148 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:10.060162 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:41:10.061450 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:10.061452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:10.061500 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:10.061577 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:10.062900 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:10.063452 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:10.064705 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:10.064920 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:10.065008 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:10.068140 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:10.068144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:10.068147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:10.068186 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:10.069811 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:10.070560 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:10.070564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:10.070566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:10.070680 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:10.071283 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:10.071287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:10.071288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:10.071570 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:10.071679 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:10.072413 (+ 734us) server_negotiation.cc:299] Negotiation successful
0925 22:41:10.072542 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":886}
I20260925 22:41:10.073590 6386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:10.059061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:46355 (local address 127.2.74.195:60025)
0925 22:41:10.059725 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:10.059791 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:10.061204 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:10.061759 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:10.061768 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:10.062181 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:10.062733 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:10.062750 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:10.063583 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:10.063585 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:10.064556 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:10.064568 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:10.065076 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:10.065079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:10.065244 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:10.065958 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:10.065977 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:10.067912 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:10.069977 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:10.069984 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:10.070007 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:10.070388 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:10.070813 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:10.070817 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:10.070819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:10.071128 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:10.071710 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:10.071719 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:10.072120 (+ 401us) client_negotiation.cc:769] Sending connection context
0925 22:41:10.072336 (+ 216us) client_negotiation.cc:240] Negotiation successful
0925 22:41:10.072650 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":171,"threads_started":1}
I20260925 22:41:10.074824 6384 heartbeater.cc:347] Connected to a master server at 127.2.74.254:46355
I20260925 22:41:10.075148 6384 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:10.075726 6384 heartbeater.cc:511] Master 127.2.74.254:46355 requested a full tablet report, sending...
I20260925 22:41:10.076855 5925 ts_manager.cc:194] Registered new tserver with Master: aa5ad0e00da9446e8d82ba2f025a412a (127.2.74.195:41477)
I20260925 22:41:10.077410 5925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.195@KRBTEST.COM'} at 127.2.74.195:60025
I20260925 22:41:10.084136 2347 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 22:41:10 dist-test-slave-csnl krb5kdc[5880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:41:10 dist-test-slave-csnl krb5kdc[5880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.193@KRBTEST.COM
I20260925 22:41:10.111382 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:10.101092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34240 (local address 127.2.74.193:41121)
0925 22:41:10.101408 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:10.101423 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:10.101450 (+ 27us) server_negotiation.cc:407] Connection header received
0925 22:41:10.101520 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:10.101529 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:10.101729 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:10.101897 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:10.102595 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:10.103500 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:10.104392 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:10.104851 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:10.104976 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:10.107639 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:10.107651 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:10.107665 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:10.107698 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:10.109554 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:10.110032 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:10.110037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:10.110038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:10.110122 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:10.110366 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:10.110368 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:10.110369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:10.110623 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:10.110811 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:10.111055 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 22:41:10.111194 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":105,"threads_started":1}
Sep 25 22:41:10 dist-test-slave-csnl krb5kdc[5880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:41:10.123576 5964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:10.114673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43952 (local address 127.2.74.254:46355)
0925 22:41:10.114814 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:10.114824 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:10.114842 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:41:10.114933 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:10.114936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:10.115040 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:10.115115 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:10.115832 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:10.116348 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:10.117158 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:10.117379 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:10.117463 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:10.120223 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:10.120227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:10.120229 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:10.120263 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:10.121916 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:10.122378 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:10.122380 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:10.122381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:10.122464 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:10.122704 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:10.122706 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:10.122707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:10.122940 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:10.123118 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:10.123337 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 22:41:10.123446 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 22:41:10.124073 5925 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43952
I20260925 22:41:10.124789 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 5987
I20260925 22:41:10.130589 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 6121
I20260925 22:41:10.136816 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 6255
I20260925 22:41:10.142328 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 5893
2026-09-25T22:41:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2175 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:41:10 dist-test-slave-csnl krb5kdc[6396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:41:10 dist-test-slave-csnl krb5kdc[6396](info): set up 2 sockets
Sep 25 22:41:10 dist-test-slave-csnl krb5kdc[6396](info): commencing operation
krb5kdc: starting...
W20260925 22:41:11.183255 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:41:11 dist-test-slave-csnl krb5kdc[6396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:41:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:41:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:41:11.308444 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:43891
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:44299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:43891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:11.424554 6409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:11.424922 6409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:11.425006 6409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:11.429323 6409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:41:11.429463 6409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:11.429502 6409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:11.429531 6409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:41:11.429570 6409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:41:11.436013 6409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:44299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:43891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:43891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:11.437227 6409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:11.438344 6409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:11.447147 6415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:11.447161 6414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:11.447201 6409 server_base.cc:1034] running on GCE node
W20260925 22:41:11.447148 6417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:11.447968 6409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:11.449159 6409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:11.450328 6409 hybrid_clock.cc:648] HybridClock initialized: now 1790376071450291 us; error 57 us; skew 500 ppm
Sep 25 22:41:11 dist-test-slave-csnl krb5kdc[6396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376071, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:11.453332 6409 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:41:11.454176 6409 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:11.454282 6409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:11.454536 6409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:11.456795 6409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "9a9eebfb901d4badbea51cf8845051b4"
format_stamp: "Formatted at 2026-09-25 22:41:11 on dist-test-slave-csnl"
server_key: "b3c5fb3b663eddaeafe56c102d4124c8"
server_key_iv: "9c4d58a76089fd6e659cbad3d05aaf78"
server_key_version: "encryptionkey@0"
I20260925 22:41:11.457399 6409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "9a9eebfb901d4badbea51cf8845051b4"
format_stamp: "Formatted at 2026-09-25 22:41:11 on dist-test-slave-csnl"
server_key: "b3c5fb3b663eddaeafe56c102d4124c8"
server_key_iv: "9c4d58a76089fd6e659cbad3d05aaf78"
server_key_version: "encryptionkey@0"
I20260925 22:41:11.461388 6409 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 22:41:11.463912 6423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:11.465227 6409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:41:11.465373 6409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "9a9eebfb901d4badbea51cf8845051b4"
format_stamp: "Formatted at 2026-09-25 22:41:11 on dist-test-slave-csnl"
server_key: "b3c5fb3b663eddaeafe56c102d4124c8"
server_key_iv: "9c4d58a76089fd6e659cbad3d05aaf78"
server_key_version: "encryptionkey@0"
I20260925 22:41:11.465497 6409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:11.488724 6409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:11.492406 6409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:11.492635 6409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:11.500205 6409 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:43891
I20260925 22:41:11.500494 6475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:43891 every 8 connection(s)
I20260925 22:41:11.501006 6409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:41:11.504050 6476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:11.505225 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 6409
I20260925 22:41:11.505331 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:41:11.505615 2347 external_mini_cluster.cc:1428] Setting key 99efd1114c14f78485cf463a076b0ee2
I20260925 22:41:11.510851 6476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a9eebfb901d4badbea51cf8845051b4: Bootstrap starting.
I20260925 22:41:11.513828 6476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a9eebfb901d4badbea51cf8845051b4: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:11.514820 6476 log.cc:824] T 00000000000000000000000000000000 P 9a9eebfb901d4badbea51cf8845051b4: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:41:11 dist-test-slave-csnl krb5kdc[6396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:41:11.517503 6476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a9eebfb901d4badbea51cf8845051b4: No bootstrap required, opened a new log
I20260925 22:41:11.521235 6479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:11.507176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39846 (local address 127.2.74.254:43891)
0925 22:41:11.507607 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:11.507630 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:11.507667 (+ 37us) server_negotiation.cc:407] Connection header received
0925 22:41:11.508308 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:11.508326 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:11.508681 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:11.508985 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:11.509925 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:11.510932 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:11.512178 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:11.512670 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:11.512793 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:11.515868 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:11.515876 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:11.515903 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:11.515932 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:11.517874 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:11.518351 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:11.518357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:11.518363 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:11.518512 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:11.518771 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:11.518774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:11.518776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:11.519338 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:11.519538 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:11.519770 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 22:41:11.520161 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":141,"threads_started":1}
I20260925 22:41:11.522374 6476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a9eebfb901d4badbea51cf8845051b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9eebfb901d4badbea51cf8845051b4" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 43891 } }
I20260925 22:41:11.522723 6476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a9eebfb901d4badbea51cf8845051b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:11.522790 6476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a9eebfb901d4badbea51cf8845051b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9eebfb901d4badbea51cf8845051b4, State: Initialized, Role: FOLLOWER
I20260925 22:41:11.523303 6476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a9eebfb901d4badbea51cf8845051b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9eebfb901d4badbea51cf8845051b4" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 43891 } }
I20260925 22:41:11.523475 6476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a9eebfb901d4badbea51cf8845051b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:41:11.523541 6476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a9eebfb901d4badbea51cf8845051b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:41:11.523644 6476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a9eebfb901d4badbea51cf8845051b4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:11.524670 6476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a9eebfb901d4badbea51cf8845051b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9eebfb901d4badbea51cf8845051b4" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 43891 } }
I20260925 22:41:11.524977 6476 leader_election.cc:302] T 00000000000000000000000000000000 P 9a9eebfb901d4badbea51cf8845051b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a9eebfb901d4badbea51cf8845051b4; no voters:
I20260925 22:41:11.525302 6476 leader_election.cc:288] T 00000000000000000000000000000000 P 9a9eebfb901d4badbea51cf8845051b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:41:11.525440 6481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a9eebfb901d4badbea51cf8845051b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:41:11.525650 6481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a9eebfb901d4badbea51cf8845051b4 [term 1 LEADER]: Becoming Leader. State: Replica: 9a9eebfb901d4badbea51cf8845051b4, State: Running, Role: LEADER
I20260925 22:41:11.525985 6481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a9eebfb901d4badbea51cf8845051b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9eebfb901d4badbea51cf8845051b4" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 43891 } }
I20260925 22:41:11.526684 6476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a9eebfb901d4badbea51cf8845051b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:41:11.527961 6482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a9eebfb901d4badbea51cf8845051b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a9eebfb901d4badbea51cf8845051b4. Latest consensus state: current_term: 1 leader_uuid: "9a9eebfb901d4badbea51cf8845051b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9eebfb901d4badbea51cf8845051b4" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 43891 } } }
I20260925 22:41:11.528108 6482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a9eebfb901d4badbea51cf8845051b4 [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:11.528497 6490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:41:11.528442 6483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a9eebfb901d4badbea51cf8845051b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a9eebfb901d4badbea51cf8845051b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9eebfb901d4badbea51cf8845051b4" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 43891 } } }
I20260925 22:41:11.528609 6483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a9eebfb901d4badbea51cf8845051b4 [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:11.532188 6490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:41:11.538565 6490 catalog_manager.cc:1347] Generated new cluster ID: cd2373ec334a44b4ac40d61736a4e6da
I20260925 22:41:11.538661 6490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:41:11.566588 6490 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:41:11.567972 6490 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:41:11.579923 6490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a9eebfb901d4badbea51cf8845051b4: Generated new TSK 0
I20260925 22:41:11.580721 6490 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
I20260925 22:41:11.621857 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:0
--local_ip_for_outbound_sockets=127.2.74.193
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:43891
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:44299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:11.740813 6502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:11.741161 6502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:11.741271 6502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:11.745731 6502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:11.745844 6502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:11.745944 6502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:41:11.751665 6502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:44299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:43891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:11.752859 6502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:11.754007 6502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:11.764228 6510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:11.764228 6507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:11.764228 6508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:11.764474 6502 server_base.cc:1034] running on GCE node
I20260925 22:41:11.765076 6502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:11.765782 6502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:11.767007 6502 hybrid_clock.cc:648] HybridClock initialized: now 1790376071766966 us; error 58 us; skew 500 ppm
Sep 25 22:41:11 dist-test-slave-csnl krb5kdc[6396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376071, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:11.770208 6502 init.cc:377] Logged in from keytab as kudu/127.2.74.193@KRBTEST.COM (short username kudu)
I20260925 22:41:11.771095 6502 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:11.771170 6502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:11.771402 6502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:11.773645 6502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-0/data/instance:
uuid: "3288c2aec4fd4ff797dd041d85a7ae99"
format_stamp: "Formatted at 2026-09-25 22:41:11 on dist-test-slave-csnl"
server_key: "d4b9035a40a6a01fbbccf9f7140d53d7"
server_key_iv: "27a1884e545b0a52517057ec28d3b1ab"
server_key_version: "encryptionkey@0"
I20260925 22:41:11.774374 6502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance:
uuid: "3288c2aec4fd4ff797dd041d85a7ae99"
format_stamp: "Formatted at 2026-09-25 22:41:11 on dist-test-slave-csnl"
server_key: "d4b9035a40a6a01fbbccf9f7140d53d7"
server_key_iv: "27a1884e545b0a52517057ec28d3b1ab"
server_key_version: "encryptionkey@0"
I20260925 22:41:11.778537 6502 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:41:11.781286 6516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:11.782785 6502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:41:11.782949 6502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "3288c2aec4fd4ff797dd041d85a7ae99"
format_stamp: "Formatted at 2026-09-25 22:41:11 on dist-test-slave-csnl"
server_key: "d4b9035a40a6a01fbbccf9f7140d53d7"
server_key_iv: "27a1884e545b0a52517057ec28d3b1ab"
server_key_version: "encryptionkey@0"
I20260925 22:41:11.783108 6502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:11.808760 6502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:11.812049 6502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:11.812259 6502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:11.812927 6502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:11.813853 6502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:11.813905 6502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:11.813975 6502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:11.814009 6502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:11.823418 6502 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:45677
I20260925 22:41:11.823434 6629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:45677 every 8 connection(s)
I20260925 22:41:11.824088 6502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:41:11.829455 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 6502
I20260925 22:41:11.829564 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance
I20260925 22:41:11.829941 2347 external_mini_cluster.cc:1428] Setting key fe9329706a8c8a3591e6d3dd3e2779fd
Sep 25 22:41:11 dist-test-slave-csnl krb5kdc[6396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376071, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:41:11.839756 6479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:11.825904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:56305 (local address 127.2.74.254:43891)
0925 22:41:11.826044 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:11.826048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:11.827080 (+ 1032us) server_negotiation.cc:407] Connection header received
0925 22:41:11.828658 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:11.828662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:11.828720 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:11.828834 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:11.830202 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:11.830738 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:11.832034 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:11.832387 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:11.832505 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:11.835503 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:11.835510 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:11.835516 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:11.835553 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:11.837323 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:11.837919 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:11.837923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:11.837927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:11.838024 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:11.838368 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:11.838374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:11.838378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:11.838651 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:11.838762 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:11.839387 (+ 625us) server_negotiation.cc:299] Negotiation successful
0925 22:41:11.839540 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.2.74.194@KRBTEST.COM; defaulting to no policy
I20260925 22:41:11.840606 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:11.826375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:43891 (local address 127.2.74.193:56305)
0925 22:41:11.826875 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:11.826922 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:11.828378 (+ 1456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:11.829029 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:11.829037 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:11.829416 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:11.830036 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:11.830056 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:11.830880 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:11.830883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:11.831895 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:11.831905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:11.832550 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:11.832554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:11.832810 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:11.833490 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:11.833518 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:11.835297 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:11.837477 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:11.837482 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:11.837499 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:11.837786 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:11.838119 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:11.838121 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:11.838122 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:11.838263 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:11.838766 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:11.838771 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:11.839116 (+ 345us) client_negotiation.cc:769] Sending connection context
0925 22:41:11.839277 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 22:41:11.839556 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":121,"threads_started":1}
I20260925 22:41:11.841974 6630 heartbeater.cc:347] Connected to a master server at 127.2.74.254:43891
I20260925 22:41:11.842303 6630 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.194@KRBTEST.COM" created.
I20260925 22:41:11.842908 6630 heartbeater.cc:511] Master 127.2.74.254:43891 requested a full tablet report, sending...
I20260925 22:41:11.844781 6440 ts_manager.cc:194] Registered new tserver with Master: 3288c2aec4fd4ff797dd041d85a7ae99 (127.2.74.193:45677)
I20260925 22:41:11.845954 6440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.193@KRBTEST.COM'} at 127.2.74.193:56305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
I20260925 22:41:11.858321 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:0
--local_ip_for_outbound_sockets=127.2.74.194
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:43891
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:44299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:11.971163 6635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:11.971477 6635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:11.971562 6635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:11.975936 6635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:11.976018 6635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:11.976117 6635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:41:11.981952 6635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:44299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:43891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:11.983239 6635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:11.984341 6635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:11.994503 6640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:11.994498 6641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:11.994498 6643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:11.995023 6635 server_base.cc:1034] running on GCE node
I20260925 22:41:11.995476 6635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:11.996181 6635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:11.997364 6635 hybrid_clock.cc:648] HybridClock initialized: now 1790376071997357 us; error 46 us; skew 500 ppm
Sep 25 22:41:11 dist-test-slave-csnl krb5kdc[6396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376071, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:12.000415 6635 init.cc:377] Logged in from keytab as kudu/127.2.74.194@KRBTEST.COM (short username kudu)
I20260925 22:41:12.001272 6635 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:12.001387 6635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:12.001636 6635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:12.003917 6635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-1/data/instance:
uuid: "947eca2781924c4bb979fead4013fac5"
format_stamp: "Formatted at 2026-09-25 22:41:11 on dist-test-slave-csnl"
server_key: "e1d24c0175be4824f634605c6d9b4464"
server_key_iv: "5bda94e933f6d0d73c8a1131e42cda37"
server_key_version: "encryptionkey@0"
I20260925 22:41:12.004551 6635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance:
uuid: "947eca2781924c4bb979fead4013fac5"
format_stamp: "Formatted at 2026-09-25 22:41:11 on dist-test-slave-csnl"
server_key: "e1d24c0175be4824f634605c6d9b4464"
server_key_iv: "5bda94e933f6d0d73c8a1131e42cda37"
server_key_version: "encryptionkey@0"
I20260925 22:41:12.008948 6635 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:41:12.011719 6649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:12.013191 6635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:41:12.013386 6635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "947eca2781924c4bb979fead4013fac5"
format_stamp: "Formatted at 2026-09-25 22:41:11 on dist-test-slave-csnl"
server_key: "e1d24c0175be4824f634605c6d9b4464"
server_key_iv: "5bda94e933f6d0d73c8a1131e42cda37"
server_key_version: "encryptionkey@0"
I20260925 22:41:12.013527 6635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:12.026602 6635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:12.030546 6635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:12.030766 6635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:12.031414 6635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:12.032260 6635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:12.032307 6635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:12.032378 6635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:12.032414 6635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:12.040926 6635 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:34401
I20260925 22:41:12.040995 6762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:34401 every 8 connection(s)
I20260925 22:41:12.041564 6635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:41:12.044898 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 6635
I20260925 22:41:12.044988 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance
I20260925 22:41:12.045254 2347 external_mini_cluster.cc:1428] Setting key cbf8662b5f94620edc1e4a7647b16e4e
Sep 25 22:41:12 dist-test-slave-csnl krb5kdc[6396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376071, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.74.195@KRBTEST.COM; defaulting to no policy
I20260925 22:41:12.056813 6479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:12.043402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:43431 (local address 127.2.74.254:43891)
0925 22:41:12.043535 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:12.043539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:12.044428 (+ 889us) server_negotiation.cc:407] Connection header received
0925 22:41:12.046057 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:12.046061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:12.046113 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:12.046189 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:12.047546 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.048072 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.049435 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.049866 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.049966 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:12.052997 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:12.053000 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:12.053003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:12.053040 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:12.054521 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:12.055133 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:12.055136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:12.055138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:12.055229 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:12.055692 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:12.055694 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:12.055696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:12.055880 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:12.055957 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:12.056557 (+ 600us) server_negotiation.cc:299] Negotiation successful
0925 22:41:12.056678 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.195@KRBTEST.COM" created.
I20260925 22:41:12.057767 6765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:12.043823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:43891 (local address 127.2.74.194:43431)
0925 22:41:12.044281 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:12.044324 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:12.045854 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:12.046363 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:12.046370 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:12.046826 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:12.047401 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:12.047417 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.048204 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.048213 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:12.049304 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:12.049314 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.050026 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.050031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:12.050218 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:12.050887 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:12.050904 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:12.052812 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:12.054657 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:12.054663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:12.054682 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:12.055018 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:12.055366 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:12.055371 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:12.055373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:12.055593 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:12.055971 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:12.055977 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:12.056322 (+ 345us) client_negotiation.cc:769] Sending connection context
0925 22:41:12.056486 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 22:41:12.056763 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":108,"threads_started":1}
I20260925 22:41:12.059068 6763 heartbeater.cc:347] Connected to a master server at 127.2.74.254:43891
I20260925 22:41:12.059342 6763 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:12.059906 6763 heartbeater.cc:511] Master 127.2.74.254:43891 requested a full tablet report, sending...
I20260925 22:41:12.061057 6440 ts_manager.cc:194] Registered new tserver with Master: 947eca2781924c4bb979fead4013fac5 (127.2.74.194:34401)
I20260925 22:41:12.061712 6440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.194@KRBTEST.COM'} at 127.2.74.194:43431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
I20260925 22:41:12.074030 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:0
--local_ip_for_outbound_sockets=127.2.74.195
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:43891
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:44299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:12.192257 6768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:12.192524 6768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:12.192571 6768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:12.196714 6768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:12.196803 6768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:12.196869 6768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:41:12.202409 6768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:44299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:43891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:12.203577 6768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:12.204598 6768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:12.214461 6773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:12.214537 6768 server_base.cc:1034] running on GCE node
W20260925 22:41:12.214461 6776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:12.214479 6774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:12.215157 6768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:12.215874 6768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:12.217044 6768 hybrid_clock.cc:648] HybridClock initialized: now 1790376072217011 us; error 53 us; skew 500 ppm
Sep 25 22:41:12 dist-test-slave-csnl krb5kdc[6396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376072, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:12.220176 6768 init.cc:377] Logged in from keytab as kudu/127.2.74.195@KRBTEST.COM (short username kudu)
I20260925 22:41:12.221127 6768 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:12.221215 6768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:12.221391 6768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:12.223603 6768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-2/data/instance:
uuid: "3ecae47c6efe4ea9bfdcec68110f3576"
format_stamp: "Formatted at 2026-09-25 22:41:12 on dist-test-slave-csnl"
server_key: "87e30350abd51e7ccbb2b5a5bbe5d82a"
server_key_iv: "863b2d8244be5c1c8457e93d6517f9bd"
server_key_version: "encryptionkey@0"
I20260925 22:41:12.224154 6768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance:
uuid: "3ecae47c6efe4ea9bfdcec68110f3576"
format_stamp: "Formatted at 2026-09-25 22:41:12 on dist-test-slave-csnl"
server_key: "87e30350abd51e7ccbb2b5a5bbe5d82a"
server_key_iv: "863b2d8244be5c1c8457e93d6517f9bd"
server_key_version: "encryptionkey@0"
I20260925 22:41:12.228222 6768 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:41:12.230683 6782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:12.232053 6768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:41:12.232184 6768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "3ecae47c6efe4ea9bfdcec68110f3576"
format_stamp: "Formatted at 2026-09-25 22:41:12 on dist-test-slave-csnl"
server_key: "87e30350abd51e7ccbb2b5a5bbe5d82a"
server_key_iv: "863b2d8244be5c1c8457e93d6517f9bd"
server_key_version: "encryptionkey@0"
I20260925 22:41:12.232299 6768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:12.261880 6768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:12.265415 6768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:12.265630 6768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:12.266253 6768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:12.267196 6768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:12.267246 6768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:12.267310 6768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:12.267343 6768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:12.277057 6768 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:46675
I20260925 22:41:12.277072 6895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:46675 every 8 connection(s)
I20260925 22:41:12.277761 6768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:41:12.281255 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 6768
I20260925 22:41:12.281421 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance
I20260925 22:41:12.281780 2347 external_mini_cluster.cc:1428] Setting key adc9297a81ff3456e1989f8f91cff200
Sep 25 22:41:12 dist-test-slave-csnl krb5kdc[6396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376072, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:41:12.292889 6479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:12.279746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:53965 (local address 127.2.74.254:43891)
0925 22:41:12.279882 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:12.279886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:12.280885 (+ 999us) server_negotiation.cc:407] Connection header received
0925 22:41:12.282683 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:12.282686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:12.282733 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:12.282796 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:12.284432 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.284960 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.286218 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.286486 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.286573 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:12.289327 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:12.289330 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:12.289333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:12.289360 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:12.290869 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:12.291448 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:12.291451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:12.291452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:12.291530 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:12.291844 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:12.291846 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:12.291847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:12.292036 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:12.292107 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:12.292652 (+ 545us) server_negotiation.cc:299] Negotiation successful
0925 22:41:12.292762 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:41:12.293814 6898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:12.280180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:43891 (local address 127.2.74.195:53965)
0925 22:41:12.280703 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:12.280769 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:12.282412 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:12.282999 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:12.283013 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:12.283560 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:12.284251 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:12.284276 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.285076 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.285079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:12.286061 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:12.286071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.286629 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.286631 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:12.286793 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:12.287506 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:12.287525 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:12.289123 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:12.291047 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:12.291053 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:12.291068 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:12.291356 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:12.291614 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:12.291617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:12.291618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:12.291755 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:12.292119 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:12.292124 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:12.292425 (+ 301us) client_negotiation.cc:769] Sending connection context
0925 22:41:12.292589 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 22:41:12.292871 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":123,"threads_started":1}
I20260925 22:41:12.295290 6896 heartbeater.cc:347] Connected to a master server at 127.2.74.254:43891
I20260925 22:41:12.295596 6896 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:12.296192 6896 heartbeater.cc:511] Master 127.2.74.254:43891 requested a full tablet report, sending...
I20260925 22:41:12.297278 6440 ts_manager.cc:194] Registered new tserver with Master: 3ecae47c6efe4ea9bfdcec68110f3576 (127.2.74.195:46675)
I20260925 22:41:12.297842 6440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.195@KRBTEST.COM'} at 127.2.74.195:53965
I20260925 22:41:12.308513 2347 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:41:12.317971 6479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:12.310644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39848 (local address 127.2.74.254:43891)
0925 22:41:12.310793 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:12.310804 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:12.310929 (+ 125us) server_negotiation.cc:407] Connection header received
0925 22:41:12.311100 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:12.311102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:12.311144 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:12.311212 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:12.311932 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.312439 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.313251 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.313469 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.313550 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:12.314893 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:12.314896 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:12.314898 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:12.314922 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:12.316454 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:12.316835 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:12.316838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:12.316839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:12.316910 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:12.317157 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:12.317159 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:12.317160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:12.317433 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:12.317538 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:12.317736 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 22:41:12.317847 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 22:41:12.321025 6440 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:41:12.324056 6440 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:41:12.339808 6912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:12.334298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46692 (local address 127.2.74.193:45677)
0925 22:41:12.334702 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:12.334709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:12.334913 (+ 204us) server_negotiation.cc:407] Connection header received
0925 22:41:12.335046 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:12.335052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:12.335252 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:12.335404 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:12.335668 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.336795 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.338131 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.339022 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.339140 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:12.339310 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:12.339513 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 22:41:12.339624 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":110,"threads_started":1}
I20260925 22:41:12.339802 6911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:12.333740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:45677 (local address 127.0.0.1:46692)
0925 22:41:12.334796 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:12.334810 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:12.334907 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:12.335396 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:12.335399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:12.335418 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:12.335545 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:12.335553 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.336955 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.336958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:12.337995 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:12.338004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.339162 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.339165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:12.339291 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:12.339333 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 22:41:12.339457 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 22:41:12.339614 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":990,"mutex_wait_us":2,"thread_start_us":94,"threads_started":1}
I20260925 22:41:12.339949 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:12.333187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59320 (local address 127.2.74.195:46675)
0925 22:41:12.333601 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:12.333615 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:12.334070 (+ 455us) server_negotiation.cc:407] Connection header received
0925 22:41:12.334328 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:12.334335 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:12.334563 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:12.334720 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:12.335258 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.336511 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.338414 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.339031 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.339162 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:12.339323 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:12.339697 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 22:41:12.339793 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":119,"threads_started":1}
I20260925 22:41:12.340171 6909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:12.333033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36108 (local address 127.2.74.194:34401)
0925 22:41:12.333780 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:12.333792 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:12.334250 (+ 458us) server_negotiation.cc:407] Connection header received
0925 22:41:12.334330 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:12.334336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:12.334566 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:12.334721 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:12.335235 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.336284 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.338393 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.339245 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.339348 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:12.339492 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:12.339891 (+ 399us) server_negotiation.cc:299] Negotiation successful
0925 22:41:12.340024 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":129,"threads_started":1}
I20260925 22:41:12.340252 6908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:12.333030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:46675 (local address 127.0.0.1:59320)
0925 22:41:12.333932 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:12.333964 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:12.334102 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:12.334925 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:12.334933 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:12.334960 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:12.335128 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:12.335135 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.336641 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.336644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:12.338257 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:12.338271 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.339384 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.339387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:12.339489 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:12.339509 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 22:41:12.339568 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 22:41:12.339629 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"mutex_wait_us":150,"thread_start_us":113,"threads_started":1}
I20260925 22:41:12.340548 6907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:12.332937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:34401 (local address 127.0.0.1:36108)
0925 22:41:12.333902 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:12.333964 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:12.334102 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:12.334925 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:12.334934 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:12.334983 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:12.335112 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:12.335117 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.336416 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.336422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:12.338253 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:12.338264 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.339553 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.339556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:12.339643 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:12.339664 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:41:12.339735 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 22:41:12.339808 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":171,"thread_start_us":106,"threads_started":1}
I20260925 22:41:12.343684 6564 tablet_service.cc:1467] Processing CreateTablet for tablet 205004c7c6c34ed986af9fd6f250a490 (DEFAULT_TABLE table=test-table [id=22f2fde6f625439d9f03f186d6387c0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:12.343637 6830 tablet_service.cc:1467] Processing CreateTablet for tablet 205004c7c6c34ed986af9fd6f250a490 (DEFAULT_TABLE table=test-table [id=22f2fde6f625439d9f03f186d6387c0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:12.343637 6697 tablet_service.cc:1467] Processing CreateTablet for tablet 205004c7c6c34ed986af9fd6f250a490 (DEFAULT_TABLE table=test-table [id=22f2fde6f625439d9f03f186d6387c0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:12.344762 6564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 205004c7c6c34ed986af9fd6f250a490. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:12.344785 6697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 205004c7c6c34ed986af9fd6f250a490. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:12.344785 6830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 205004c7c6c34ed986af9fd6f250a490. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:12.352231 6913 tablet_bootstrap.cc:492] T 205004c7c6c34ed986af9fd6f250a490 P 3288c2aec4fd4ff797dd041d85a7ae99: Bootstrap starting.
I20260925 22:41:12.353214 6915 tablet_bootstrap.cc:492] T 205004c7c6c34ed986af9fd6f250a490 P 3ecae47c6efe4ea9bfdcec68110f3576: Bootstrap starting.
I20260925 22:41:12.354472 6913 tablet_bootstrap.cc:654] T 205004c7c6c34ed986af9fd6f250a490 P 3288c2aec4fd4ff797dd041d85a7ae99: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:12.355165 6915 tablet_bootstrap.cc:654] T 205004c7c6c34ed986af9fd6f250a490 P 3ecae47c6efe4ea9bfdcec68110f3576: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:12.355372 6914 tablet_bootstrap.cc:492] T 205004c7c6c34ed986af9fd6f250a490 P 947eca2781924c4bb979fead4013fac5: Bootstrap starting.
I20260925 22:41:12.355391 6913 log.cc:824] T 205004c7c6c34ed986af9fd6f250a490 P 3288c2aec4fd4ff797dd041d85a7ae99: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:12.356586 6915 log.cc:824] T 205004c7c6c34ed986af9fd6f250a490 P 3ecae47c6efe4ea9bfdcec68110f3576: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:12.357522 6913 tablet_bootstrap.cc:492] T 205004c7c6c34ed986af9fd6f250a490 P 3288c2aec4fd4ff797dd041d85a7ae99: No bootstrap required, opened a new log
I20260925 22:41:12.357746 6914 tablet_bootstrap.cc:654] T 205004c7c6c34ed986af9fd6f250a490 P 947eca2781924c4bb979fead4013fac5: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:12.359045 6915 tablet_bootstrap.cc:492] T 205004c7c6c34ed986af9fd6f250a490 P 3ecae47c6efe4ea9bfdcec68110f3576: No bootstrap required, opened a new log
I20260925 22:41:12.359133 6913 ts_tablet_manager.cc:1397] T 205004c7c6c34ed986af9fd6f250a490 P 3288c2aec4fd4ff797dd041d85a7ae99: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:41:12.359180 6914 log.cc:824] T 205004c7c6c34ed986af9fd6f250a490 P 947eca2781924c4bb979fead4013fac5: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:12.359338 6915 ts_tablet_manager.cc:1397] T 205004c7c6c34ed986af9fd6f250a490 P 3ecae47c6efe4ea9bfdcec68110f3576: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:41:12.361596 6914 tablet_bootstrap.cc:492] T 205004c7c6c34ed986af9fd6f250a490 P 947eca2781924c4bb979fead4013fac5: No bootstrap required, opened a new log
I20260925 22:41:12.361824 6914 ts_tablet_manager.cc:1397] T 205004c7c6c34ed986af9fd6f250a490 P 947eca2781924c4bb979fead4013fac5: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 22:41:12.363133 6913 raft_consensus.cc:348] T 205004c7c6c34ed986af9fd6f250a490 P 3288c2aec4fd4ff797dd041d85a7ae99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "947eca2781924c4bb979fead4013fac5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 34401 } } peers { permanent_uuid: "3ecae47c6efe4ea9bfdcec68110f3576" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 46675 } } peers { permanent_uuid: "3288c2aec4fd4ff797dd041d85a7ae99" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 45677 } }
I20260925 22:41:12.363405 6913 raft_consensus.cc:374] T 205004c7c6c34ed986af9fd6f250a490 P 3288c2aec4fd4ff797dd041d85a7ae99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:12.363488 6913 raft_consensus.cc:729] T 205004c7c6c34ed986af9fd6f250a490 P 3288c2aec4fd4ff797dd041d85a7ae99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3288c2aec4fd4ff797dd041d85a7ae99, State: Initialized, Role: FOLLOWER
I20260925 22:41:12.363596 6915 raft_consensus.cc:348] T 205004c7c6c34ed986af9fd6f250a490 P 3ecae47c6efe4ea9bfdcec68110f3576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "947eca2781924c4bb979fead4013fac5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 34401 } } peers { permanent_uuid: "3ecae47c6efe4ea9bfdcec68110f3576" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 46675 } } peers { permanent_uuid: "3288c2aec4fd4ff797dd041d85a7ae99" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 45677 } }
I20260925 22:41:12.363924 6915 raft_consensus.cc:374] T 205004c7c6c34ed986af9fd6f250a490 P 3ecae47c6efe4ea9bfdcec68110f3576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:12.364012 6915 raft_consensus.cc:729] T 205004c7c6c34ed986af9fd6f250a490 P 3ecae47c6efe4ea9bfdcec68110f3576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ecae47c6efe4ea9bfdcec68110f3576, State: Initialized, Role: FOLLOWER
I20260925 22:41:12.364048 6913 consensus_queue.cc:260] T 205004c7c6c34ed986af9fd6f250a490 P 3288c2aec4fd4ff797dd041d85a7ae99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "947eca2781924c4bb979fead4013fac5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 34401 } } peers { permanent_uuid: "3ecae47c6efe4ea9bfdcec68110f3576" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 46675 } } peers { permanent_uuid: "3288c2aec4fd4ff797dd041d85a7ae99" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 45677 } }
I20260925 22:41:12.364490 6915 consensus_queue.cc:260] T 205004c7c6c34ed986af9fd6f250a490 P 3ecae47c6efe4ea9bfdcec68110f3576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "947eca2781924c4bb979fead4013fac5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 34401 } } peers { permanent_uuid: "3ecae47c6efe4ea9bfdcec68110f3576" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 46675 } } peers { permanent_uuid: "3288c2aec4fd4ff797dd041d85a7ae99" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 45677 } }
I20260925 22:41:12.364817 6630 heartbeater.cc:503] Master 127.2.74.254:43891 was elected leader, sending a full tablet report...
I20260925 22:41:12.365237 6913 ts_tablet_manager.cc:1428] T 205004c7c6c34ed986af9fd6f250a490 P 3288c2aec4fd4ff797dd041d85a7ae99: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:41:12.365692 6915 ts_tablet_manager.cc:1428] T 205004c7c6c34ed986af9fd6f250a490 P 3ecae47c6efe4ea9bfdcec68110f3576: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:41:12.366046 6896 heartbeater.cc:503] Master 127.2.74.254:43891 was elected leader, sending a full tablet report...
I20260925 22:41:12.366595 6914 raft_consensus.cc:348] T 205004c7c6c34ed986af9fd6f250a490 P 947eca2781924c4bb979fead4013fac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "947eca2781924c4bb979fead4013fac5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 34401 } } peers { permanent_uuid: "3ecae47c6efe4ea9bfdcec68110f3576" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 46675 } } peers { permanent_uuid: "3288c2aec4fd4ff797dd041d85a7ae99" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 45677 } }
I20260925 22:41:12.366859 6914 raft_consensus.cc:374] T 205004c7c6c34ed986af9fd6f250a490 P 947eca2781924c4bb979fead4013fac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:12.366926 6914 raft_consensus.cc:729] T 205004c7c6c34ed986af9fd6f250a490 P 947eca2781924c4bb979fead4013fac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 947eca2781924c4bb979fead4013fac5, State: Initialized, Role: FOLLOWER
I20260925 22:41:12.367465 6914 consensus_queue.cc:260] T 205004c7c6c34ed986af9fd6f250a490 P 947eca2781924c4bb979fead4013fac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "947eca2781924c4bb979fead4013fac5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 34401 } } peers { permanent_uuid: "3ecae47c6efe4ea9bfdcec68110f3576" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 46675 } } peers { permanent_uuid: "3288c2aec4fd4ff797dd041d85a7ae99" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 45677 } }
I20260925 22:41:12.368235 6763 heartbeater.cc:503] Master 127.2.74.254:43891 was elected leader, sending a full tablet report...
I20260925 22:41:12.368693 6914 ts_tablet_manager.cc:1428] T 205004c7c6c34ed986af9fd6f250a490 P 947eca2781924c4bb979fead4013fac5: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 22:41:12.380460 6919 raft_consensus.cc:482] T 205004c7c6c34ed986af9fd6f250a490 P 3288c2aec4fd4ff797dd041d85a7ae99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:41:12.380697 6919 raft_consensus.cc:504] T 205004c7c6c34ed986af9fd6f250a490 P 3288c2aec4fd4ff797dd041d85a7ae99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "947eca2781924c4bb979fead4013fac5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 34401 } } peers { permanent_uuid: "3ecae47c6efe4ea9bfdcec68110f3576" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 46675 } } peers { permanent_uuid: "3288c2aec4fd4ff797dd041d85a7ae99" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 45677 } }
I20260925 22:41:12.381884 6919 leader_election.cc:288] T 205004c7c6c34ed986af9fd6f250a490 P 3288c2aec4fd4ff797dd041d85a7ae99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 947eca2781924c4bb979fead4013fac5 (127.2.74.194:34401), 3ecae47c6efe4ea9bfdcec68110f3576 (127.2.74.195:46675)
I20260925 22:41:12.387643 6909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:12.382264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:36803 (local address 127.2.74.194:34401)
0925 22:41:12.382397 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:12.382402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:12.382801 (+ 399us) server_negotiation.cc:407] Connection header received
0925 22:41:12.383103 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:12.383106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:12.383203 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:12.383315 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:12.383732 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.384547 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.386186 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.387035 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.387181 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:12.387240 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:12.387427 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 22:41:12.387500 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:41:12.387717 6922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:12.382190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:34401 (local address 127.2.74.193:36803)
0925 22:41:12.382695 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:12.382713 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:12.382904 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:12.383361 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:12.383364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:12.383384 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:12.383588 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:12.383596 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.384699 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.384702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:12.386011 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:12.386043 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.387193 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.387197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:12.387306 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:12.387331 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:41:12.387411 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 22:41:12.387495 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":468,"threads_started":1}
I20260925 22:41:12.388008 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:12.382250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:59057 (local address 127.2.74.195:46675)
0925 22:41:12.382397 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:12.382402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:12.383087 (+ 685us) server_negotiation.cc:407] Connection header received
0925 22:41:12.383288 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:12.383293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:12.383351 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:12.383416 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:12.383860 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.384692 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.386426 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.387202 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.387318 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:12.387378 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:12.387752 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 22:41:12.387862 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 22:41:12.388528 6923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:12.382190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:46675 (local address 127.2.74.193:59057)
0925 22:41:12.382980 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:12.382998 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:12.383111 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:12.383526 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:12.383531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:12.383553 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:12.383701 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:12.383708 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.384860 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.384866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:12.386257 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:12.386270 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.387481 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.387485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:12.387598 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:12.387621 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:41:12.388242 (+ 621us) client_negotiation.cc:240] Negotiation successful
0925 22:41:12.388346 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":30,"thread_start_us":125,"threads_started":1}
I20260925 22:41:12.389246 6717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205004c7c6c34ed986af9fd6f250a490" candidate_uuid: "3288c2aec4fd4ff797dd041d85a7ae99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "947eca2781924c4bb979fead4013fac5" is_pre_election: true
I20260925 22:41:12.389632 6717 raft_consensus.cc:2454] T 205004c7c6c34ed986af9fd6f250a490 P 947eca2781924c4bb979fead4013fac5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3288c2aec4fd4ff797dd041d85a7ae99 in term 0.
I20260925 22:41:12.390240 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205004c7c6c34ed986af9fd6f250a490" candidate_uuid: "3288c2aec4fd4ff797dd041d85a7ae99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ecae47c6efe4ea9bfdcec68110f3576" is_pre_election: true
I20260925 22:41:12.390300 6518 leader_election.cc:302] T 205004c7c6c34ed986af9fd6f250a490 P 3288c2aec4fd4ff797dd041d85a7ae99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3288c2aec4fd4ff797dd041d85a7ae99, 947eca2781924c4bb979fead4013fac5; no voters:
I20260925 22:41:12.390597 6850 raft_consensus.cc:2454] T 205004c7c6c34ed986af9fd6f250a490 P 3ecae47c6efe4ea9bfdcec68110f3576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3288c2aec4fd4ff797dd041d85a7ae99 in term 0.
I20260925 22:41:12.390663 6919 raft_consensus.cc:2781] T 205004c7c6c34ed986af9fd6f250a490 P 3288c2aec4fd4ff797dd041d85a7ae99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:41:12.390795 6919 raft_consensus.cc:482] T 205004c7c6c34ed986af9fd6f250a490 P 3288c2aec4fd4ff797dd041d85a7ae99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:41:12.390862 6919 raft_consensus.cc:3037] T 205004c7c6c34ed986af9fd6f250a490 P 3288c2aec4fd4ff797dd041d85a7ae99 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:12.392195 6919 raft_consensus.cc:504] T 205004c7c6c34ed986af9fd6f250a490 P 3288c2aec4fd4ff797dd041d85a7ae99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "947eca2781924c4bb979fead4013fac5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 34401 } } peers { permanent_uuid: "3ecae47c6efe4ea9bfdcec68110f3576" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 46675 } } peers { permanent_uuid: "3288c2aec4fd4ff797dd041d85a7ae99" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 45677 } }
I20260925 22:41:12.392632 6919 leader_election.cc:288] T 205004c7c6c34ed986af9fd6f250a490 P 3288c2aec4fd4ff797dd041d85a7ae99 [CANDIDATE]: Term 1 election: Requested vote from peers 947eca2781924c4bb979fead4013fac5 (127.2.74.194:34401), 3ecae47c6efe4ea9bfdcec68110f3576 (127.2.74.195:46675)
I20260925 22:41:12.393045 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205004c7c6c34ed986af9fd6f250a490" candidate_uuid: "3288c2aec4fd4ff797dd041d85a7ae99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ecae47c6efe4ea9bfdcec68110f3576"
I20260925 22:41:12.393165 6850 raft_consensus.cc:3037] T 205004c7c6c34ed986af9fd6f250a490 P 3ecae47c6efe4ea9bfdcec68110f3576 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:12.394644 6850 raft_consensus.cc:2454] T 205004c7c6c34ed986af9fd6f250a490 P 3ecae47c6efe4ea9bfdcec68110f3576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3288c2aec4fd4ff797dd041d85a7ae99 in term 1.
I20260925 22:41:12.395056 6717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205004c7c6c34ed986af9fd6f250a490" candidate_uuid: "3288c2aec4fd4ff797dd041d85a7ae99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "947eca2781924c4bb979fead4013fac5"
I20260925 22:41:12.395098 6519 leader_election.cc:302] T 205004c7c6c34ed986af9fd6f250a490 P 3288c2aec4fd4ff797dd041d85a7ae99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3288c2aec4fd4ff797dd041d85a7ae99, 3ecae47c6efe4ea9bfdcec68110f3576; no voters:
I20260925 22:41:12.395208 6717 raft_consensus.cc:3037] T 205004c7c6c34ed986af9fd6f250a490 P 947eca2781924c4bb979fead4013fac5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:12.395306 6919 raft_consensus.cc:2781] T 205004c7c6c34ed986af9fd6f250a490 P 3288c2aec4fd4ff797dd041d85a7ae99 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:41:12.395555 6919 raft_consensus.cc:686] T 205004c7c6c34ed986af9fd6f250a490 P 3288c2aec4fd4ff797dd041d85a7ae99 [term 1 LEADER]: Becoming Leader. State: Replica: 3288c2aec4fd4ff797dd041d85a7ae99, State: Running, Role: LEADER
I20260925 22:41:12.395925 6919 consensus_queue.cc:237] T 205004c7c6c34ed986af9fd6f250a490 P 3288c2aec4fd4ff797dd041d85a7ae99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "947eca2781924c4bb979fead4013fac5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 34401 } } peers { permanent_uuid: "3ecae47c6efe4ea9bfdcec68110f3576" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 46675 } } peers { permanent_uuid: "3288c2aec4fd4ff797dd041d85a7ae99" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 45677 } }
I20260925 22:41:12.396603 6717 raft_consensus.cc:2454] T 205004c7c6c34ed986af9fd6f250a490 P 947eca2781924c4bb979fead4013fac5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3288c2aec4fd4ff797dd041d85a7ae99 in term 1.
I20260925 22:41:12.399686 6439 catalog_manager.cc:5495] T 205004c7c6c34ed986af9fd6f250a490 P 3288c2aec4fd4ff797dd041d85a7ae99 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3288c2aec4fd4ff797dd041d85a7ae99 (127.2.74.193). New cstate: current_term: 1 leader_uuid: "3288c2aec4fd4ff797dd041d85a7ae99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "947eca2781924c4bb979fead4013fac5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 34401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ecae47c6efe4ea9bfdcec68110f3576" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 46675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3288c2aec4fd4ff797dd041d85a7ae99" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 45677 } health_report { overall_health: HEALTHY } } }
I20260925 22:41:12.422441 6912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:12.418802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46706 (local address 127.2.74.193:45677)
0925 22:41:12.418930 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:12.418935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:12.418971 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:41:12.419154 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:12.419156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:12.419216 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:12.419326 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:41:12.419583 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.420082 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.420948 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:12.421168 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:12.421290 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:12.421600 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:41:12.422022 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:41:12.422154 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:12.422214 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 22:41:12.422275 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 22:41:12.428761 6717 raft_consensus.cc:1261] T 205004c7c6c34ed986af9fd6f250a490 P 947eca2781924c4bb979fead4013fac5 [term 1 FOLLOWER]: Refusing update from remote peer 3288c2aec4fd4ff797dd041d85a7ae99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:41:12.428761 6850 raft_consensus.cc:1261] T 205004c7c6c34ed986af9fd6f250a490 P 3ecae47c6efe4ea9bfdcec68110f3576 [term 1 FOLLOWER]: Refusing update from remote peer 3288c2aec4fd4ff797dd041d85a7ae99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:41:12.429528 6919 consensus_queue.cc:1035] T 205004c7c6c34ed986af9fd6f250a490 P 3288c2aec4fd4ff797dd041d85a7ae99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "947eca2781924c4bb979fead4013fac5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:41:12.429790 6924 consensus_queue.cc:1035] T 205004c7c6c34ed986af9fd6f250a490 P 3288c2aec4fd4ff797dd041d85a7ae99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ecae47c6efe4ea9bfdcec68110f3576" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 46675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:41:12.439073 6928 mvcc.cc:205] Tried to move back new op lower bound from 7333380392659693568 to 7333380392536678400. Current Snapshot: MvccSnapshot[applied={T|T < 7333380392659693568}]
I20260925 22:41:12.447098 6439 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:41:12.447269 6439 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:41:12.449976 6439 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a9eebfb901d4badbea51cf8845051b4: Sending DeleteTablet for 3 replicas of tablet 205004c7c6c34ed986af9fd6f250a490
I20260925 22:41:12.450935 6830 tablet_service.cc:1514] Processing DeleteTablet for tablet 205004c7c6c34ed986af9fd6f250a490 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:12 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:59320
I20260925 22:41:12.451056 6697 tablet_service.cc:1514] Processing DeleteTablet for tablet 205004c7c6c34ed986af9fd6f250a490 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:12 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:36108
I20260925 22:41:12.451071 6564 tablet_service.cc:1514] Processing DeleteTablet for tablet 205004c7c6c34ed986af9fd6f250a490 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:12 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:46692
I20260925 22:41:12.451555 6937 tablet_replica.cc:330] T 205004c7c6c34ed986af9fd6f250a490 P 947eca2781924c4bb979fead4013fac5: stopping tablet replica
I20260925 22:41:12.451799 6938 tablet_replica.cc:330] T 205004c7c6c34ed986af9fd6f250a490 P 3288c2aec4fd4ff797dd041d85a7ae99: stopping tablet replica
I20260925 22:41:12.452136 6938 raft_consensus.cc:2229] T 205004c7c6c34ed986af9fd6f250a490 P 3288c2aec4fd4ff797dd041d85a7ae99 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:41:12.452280 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 6502
I20260925 22:41:12.454180 6937 raft_consensus.cc:2229] T 205004c7c6c34ed986af9fd6f250a490 P 947eca2781924c4bb979fead4013fac5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:41:12.454656 6936 tablet_replica.cc:330] T 205004c7c6c34ed986af9fd6f250a490 P 3ecae47c6efe4ea9bfdcec68110f3576: stopping tablet replica
I20260925 22:41:12.454670 6937 raft_consensus.cc:2258] T 205004c7c6c34ed986af9fd6f250a490 P 947eca2781924c4bb979fead4013fac5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:41:12.455061 6936 raft_consensus.cc:2229] T 205004c7c6c34ed986af9fd6f250a490 P 3ecae47c6efe4ea9bfdcec68110f3576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:41:12.455400 6936 raft_consensus.cc:2258] T 205004c7c6c34ed986af9fd6f250a490 P 3ecae47c6efe4ea9bfdcec68110f3576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:41:12.456754 6936 ts_tablet_manager.cc:1905] T 205004c7c6c34ed986af9fd6f250a490 P 3ecae47c6efe4ea9bfdcec68110f3576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:41:12.456755 6937 ts_tablet_manager.cc:1905] T 205004c7c6c34ed986af9fd6f250a490 P 947eca2781924c4bb979fead4013fac5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:41:12.459267 6936 ts_tablet_manager.cc:1918] T 205004c7c6c34ed986af9fd6f250a490 P 3ecae47c6efe4ea9bfdcec68110f3576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:41:12.459412 6936 log.cc:1196] T 205004c7c6c34ed986af9fd6f250a490 P 3ecae47c6efe4ea9bfdcec68110f3576: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-2/wal/wals/205004c7c6c34ed986af9fd6f250a490
I20260925 22:41:12.459720 6936 ts_tablet_manager.cc:1939] T 205004c7c6c34ed986af9fd6f250a490 P 3ecae47c6efe4ea9bfdcec68110f3576: Deleting consensus metadata
I20260925 22:41:12.460270 6937 ts_tablet_manager.cc:1918] T 205004c7c6c34ed986af9fd6f250a490 P 947eca2781924c4bb979fead4013fac5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260925 22:41:12.460335 6425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.74.193:45677 (error 108)
I20260925 22:41:12.460386 6937 log.cc:1196] T 205004c7c6c34ed986af9fd6f250a490 P 947eca2781924c4bb979fead4013fac5: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376047580784-2347-0/minicluster-data/ts-1/wal/wals/205004c7c6c34ed986af9fd6f250a490
I20260925 22:41:12.460752 6937 ts_tablet_manager.cc:1939] T 205004c7c6c34ed986af9fd6f250a490 P 947eca2781924c4bb979fead4013fac5: Deleting consensus metadata
I20260925 22:41:12.461236 6426 catalog_manager.cc:4841] TS 3ecae47c6efe4ea9bfdcec68110f3576 (127.2.74.195:46675): tablet 205004c7c6c34ed986af9fd6f250a490 (table test-table [id=22f2fde6f625439d9f03f186d6387c0c]) successfully deleted
I20260925 22:41:12.461696 6907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:12.461378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:45677 (local address 127.0.0.1:46722)
0925 22:41:12.461502 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:12.461573 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.193:45677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 22:41:12.461733 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 6635
W20260925 22:41:12.461898 6425 catalog_manager.cc:4584] TS 3288c2aec4fd4ff797dd041d85a7ae99 (127.2.74.193:45677): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 205004c7c6c34ed986af9fd6f250a490: Network error: Client connection negotiation failed: client connection to 127.2.74.193:45677: connect: Connection refused (error 111)
I20260925 22:41:12.467711 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 6768
I20260925 22:41:12.468811 6907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:12.468470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:34401 (local address 127.0.0.1:36112)
0925 22:41:12.468607 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:12.468689 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.194:34401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 22:41:12.473958 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 6409
2026-09-25T22:41:12Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2331 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T22:41:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:41:12Z Disabled control of system clock
I20260925 22:41:12.504757 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:43731
--webserver_interface=127.2.74.254
--webserver_port=0
--builtin_ntp_servers=127.2.74.212:45611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:43731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 22:41:12.619086 6943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:12.619396 6943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:12.619446 6943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:12.623912 6943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:41:12.623996 6943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:12.624011 6943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:12.624030 6943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:41:12.624045 6943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:41:12.629127 6943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:45611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:43731
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:43731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:12.630470 6943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:12.632033 6943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:12.641041 6948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:12.641041 6951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:12.641353 6943 server_base.cc:1034] running on GCE node
W20260925 22:41:12.641109 6949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:12.642174 6943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:12.643544 6943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:12.644733 6943 hybrid_clock.cc:648] HybridClock initialized: now 1790376072644717 us; error 33 us; skew 500 ppm
I20260925 22:41:12.656529 6943 webserver.cc:460] Webserver started at http://127.2.74.254:44927/ using document root <none> and password file <none>
I20260925 22:41:12.657231 6943 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:12.657343 6943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:12.657596 6943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:12.659964 6943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "5a83abb7e9d04896b926926010828cfe"
format_stamp: "Formatted at 2026-09-25 22:41:12 on dist-test-slave-csnl"
server_key: "cc51b6590f9306f0521c9b80e2a2ade2"
server_key_iv: "7bfbc01c11ea8cb66bffc05a70681154"
server_key_version: "encryptionkey@0"
I20260925 22:41:12.660648 6943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "5a83abb7e9d04896b926926010828cfe"
format_stamp: "Formatted at 2026-09-25 22:41:12 on dist-test-slave-csnl"
server_key: "cc51b6590f9306f0521c9b80e2a2ade2"
server_key_iv: "7bfbc01c11ea8cb66bffc05a70681154"
server_key_version: "encryptionkey@0"
I20260925 22:41:12.664852 6943 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:41:12.667703 6957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:12.669056 6943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:41:12.669232 6943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "5a83abb7e9d04896b926926010828cfe"
format_stamp: "Formatted at 2026-09-25 22:41:12 on dist-test-slave-csnl"
server_key: "cc51b6590f9306f0521c9b80e2a2ade2"
server_key_iv: "7bfbc01c11ea8cb66bffc05a70681154"
server_key_version: "encryptionkey@0"
I20260925 22:41:12.669371 6943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:12.674573 6943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:12.675343 6943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:12.675552 6943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:12.682813 6943 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:43731
I20260925 22:41:12.682828 7009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:43731 every 8 connection(s)
I20260925 22:41:12.683889 6943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:41:12.687304 7010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:12.691251 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 6943
I20260925 22:41:12.691404 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:41:12.691672 2347 external_mini_cluster.cc:1428] Setting key e67b9c7325b92cda7836b1aac88887c8
I20260925 22:41:12.693776 7010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a83abb7e9d04896b926926010828cfe: Bootstrap starting.
I20260925 22:41:12.696601 7010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a83abb7e9d04896b926926010828cfe: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:12.697461 7010 log.cc:824] T 00000000000000000000000000000000 P 5a83abb7e9d04896b926926010828cfe: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:12.697875 7013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:12.693221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35696 (local address 127.2.74.254:43731)
0925 22:41:12.693887 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:12.693903 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:12.693981 (+ 78us) server_negotiation.cc:407] Connection header received
0925 22:41:12.694676 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:12.694695 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:12.695184 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:12.695509 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:12.696398 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:12.696411 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:12.696424 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:41:12.696437 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:12.696495 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:41:12.696511 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:12.696600 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:12.696782 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 22:41:12.697083 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":146,"threads_started":1}
I20260925 22:41:12.699652 7010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a83abb7e9d04896b926926010828cfe: No bootstrap required, opened a new log
I20260925 22:41:12.703430 7010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a83abb7e9d04896b926926010828cfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a83abb7e9d04896b926926010828cfe" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 43731 } }
I20260925 22:41:12.703754 7010 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a83abb7e9d04896b926926010828cfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:12.703830 7010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a83abb7e9d04896b926926010828cfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a83abb7e9d04896b926926010828cfe, State: Initialized, Role: FOLLOWER
I20260925 22:41:12.704339 7010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a83abb7e9d04896b926926010828cfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a83abb7e9d04896b926926010828cfe" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 43731 } }
I20260925 22:41:12.704502 7010 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a83abb7e9d04896b926926010828cfe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:41:12.704550 7010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a83abb7e9d04896b926926010828cfe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:41:12.704625 7010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a83abb7e9d04896b926926010828cfe [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:12.705545 7010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a83abb7e9d04896b926926010828cfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a83abb7e9d04896b926926010828cfe" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 43731 } }
I20260925 22:41:12.705852 7010 leader_election.cc:302] T 00000000000000000000000000000000 P 5a83abb7e9d04896b926926010828cfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a83abb7e9d04896b926926010828cfe; no voters:
I20260925 22:41:12.706112 7010 leader_election.cc:288] T 00000000000000000000000000000000 P 5a83abb7e9d04896b926926010828cfe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:41:12.706250 7015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a83abb7e9d04896b926926010828cfe [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:41:12.706524 7015 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a83abb7e9d04896b926926010828cfe [term 1 LEADER]: Becoming Leader. State: Replica: 5a83abb7e9d04896b926926010828cfe, State: Running, Role: LEADER
I20260925 22:41:12.706841 7015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a83abb7e9d04896b926926010828cfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a83abb7e9d04896b926926010828cfe" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 43731 } }
I20260925 22:41:12.707398 7010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a83abb7e9d04896b926926010828cfe [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:41:12.708501 7016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a83abb7e9d04896b926926010828cfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a83abb7e9d04896b926926010828cfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a83abb7e9d04896b926926010828cfe" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 43731 } } }
I20260925 22:41:12.708650 7016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a83abb7e9d04896b926926010828cfe [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:12.708986 7017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a83abb7e9d04896b926926010828cfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a83abb7e9d04896b926926010828cfe. Latest consensus state: current_term: 1 leader_uuid: "5a83abb7e9d04896b926926010828cfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a83abb7e9d04896b926926010828cfe" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 43731 } } }
I20260925 22:41:12.709097 7017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a83abb7e9d04896b926926010828cfe [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:12.709052 7024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:41:12.712338 7024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:41:12.717741 7024 catalog_manager.cc:1347] Generated new cluster ID: 23e5bff564f5447ab6f6702daeb3c8ee
I20260925 22:41:12.717828 7024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:41:12.727259 7024 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:41:12.727634 7024 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:41:12.733865 7024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a83abb7e9d04896b926926010828cfe: Generated new TSK 0
I20260925 22:41:12.734680 7024 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:41:12.740896 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:0
--local_ip_for_outbound_sockets=127.2.74.193
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:43731
--builtin_ntp_servers=127.2.74.212:45611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 22:41:12.860299 7034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:12.860643 7034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:12.860733 7034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:12.865024 7034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:12.865125 7034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:12.865219 7034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:41:12.870425 7034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:45611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:43731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:12.871630 7034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:12.873178 7034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:12.882902 7039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:12.882915 7042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:12.882926 7040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:12.883159 7034 server_base.cc:1034] running on GCE node
I20260925 22:41:12.883658 7034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:12.884375 7034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:12.885569 7034 hybrid_clock.cc:648] HybridClock initialized: now 1790376072885556 us; error 36 us; skew 500 ppm
I20260925 22:41:12.887493 7034 webserver.cc:460] Webserver started at http://127.2.74.193:35377/ using document root <none> and password file <none>
I20260925 22:41:12.888103 7034 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:12.888199 7034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:12.888444 7034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:12.890584 7034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-0/data/instance:
uuid: "c9ac3ee73cc445fca965f090d050f4f9"
format_stamp: "Formatted at 2026-09-25 22:41:12 on dist-test-slave-csnl"
server_key: "ed09e73c0f6666cc07aa9c4e5ec1f93e"
server_key_iv: "879f99a4ec09a2739973b447f0a9ebad"
server_key_version: "encryptionkey@0"
I20260925 22:41:12.891214 7034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance:
uuid: "c9ac3ee73cc445fca965f090d050f4f9"
format_stamp: "Formatted at 2026-09-25 22:41:12 on dist-test-slave-csnl"
server_key: "ed09e73c0f6666cc07aa9c4e5ec1f93e"
server_key_iv: "879f99a4ec09a2739973b447f0a9ebad"
server_key_version: "encryptionkey@0"
I20260925 22:41:12.895421 7034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:41:12.898128 7048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:12.899621 7034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:41:12.899784 7034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "c9ac3ee73cc445fca965f090d050f4f9"
format_stamp: "Formatted at 2026-09-25 22:41:12 on dist-test-slave-csnl"
server_key: "ed09e73c0f6666cc07aa9c4e5ec1f93e"
server_key_iv: "879f99a4ec09a2739973b447f0a9ebad"
server_key_version: "encryptionkey@0"
I20260925 22:41:12.899922 7034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:12.904793 7034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:12.905457 7034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:12.905658 7034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:12.906255 7034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:12.907316 7034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:12.907383 7034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:12.907455 7034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:12.907508 7034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:12.915967 7034 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:39735
I20260925 22:41:12.915990 7161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:39735 every 8 connection(s)
I20260925 22:41:12.916899 7034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:41:12.917361 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 7034
I20260925 22:41:12.917487 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance
I20260925 22:41:12.917804 2347 external_mini_cluster.cc:1428] Setting key c723cd16254c4ce62d80b66474ebd314
I20260925 22:41:12.919890 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:0
--local_ip_for_outbound_sockets=127.2.74.194
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:43731
--builtin_ntp_servers=127.2.74.212:45611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 22:41:12.924026 7013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:12.918918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:40961 (local address 127.2.74.254:43731)
0925 22:41:12.919066 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:12.919070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:12.920408 (+ 1338us) server_negotiation.cc:407] Connection header received
0925 22:41:12.922045 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:12.922048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:12.922085 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:12.922156 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:12.923537 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:12.923540 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:12.923542 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:41:12.923553 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:12.923572 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:41:12.923580 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:12.923660 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:12.923854 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 22:41:12.923903 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 22:41:12.925333 7164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:12.919511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:43731 (local address 127.2.74.193:40961)
0925 22:41:12.920239 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:12.920292 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:12.921779 (+ 1487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:12.922367 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:12.922379 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:12.922497 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:12.923206 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:41:12.923210 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:12.923268 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:41:12.923362 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:12.923672 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:12.923684 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:12.923693 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 22:41:12.923889 (+ 196us) client_negotiation.cc:240] Negotiation successful
0925 22:41:12.924190 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":131,"threads_started":1}
I20260925 22:41:12.926789 7162 heartbeater.cc:347] Connected to a master server at 127.2.74.254:43731
I20260925 22:41:12.927125 7162 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:12.927615 7162 heartbeater.cc:511] Master 127.2.74.254:43731 requested a full tablet report, sending...
I20260925 22:41:12.929183 6974 ts_manager.cc:194] Registered new tserver with Master: c9ac3ee73cc445fca965f090d050f4f9 (127.2.74.193:39735)
W20260925 22:41:13.040495 7165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:13.040858 7165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:13.040936 7165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:13.045325 7165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:13.045451 7165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:13.045563 7165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:41:13.050865 7165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:45611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:43731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:13.052173 7165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:13.053759 7165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:13.064402 7170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:13.064392 7171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:13.064389 7165 server_base.cc:1034] running on GCE node
W20260925 22:41:13.064452 7173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:13.065028 7165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:13.065696 7165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:13.066857 7165 hybrid_clock.cc:648] HybridClock initialized: now 1790376073066842 us; error 32 us; skew 500 ppm
I20260925 22:41:13.068868 7165 webserver.cc:460] Webserver started at http://127.2.74.194:33867/ using document root <none> and password file <none>
I20260925 22:41:13.069533 7165 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:13.069602 7165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:13.069820 7165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:13.072198 7165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-1/data/instance:
uuid: "6bde3f645827492d889d57c9f3f475fc"
format_stamp: "Formatted at 2026-09-25 22:41:13 on dist-test-slave-csnl"
server_key: "5f95d28a3afe2b9c1f8e5d13753c85ea"
server_key_iv: "c6fc485c35d2d41098b975f3e6a710eb"
server_key_version: "encryptionkey@0"
I20260925 22:41:13.072861 7165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance:
uuid: "6bde3f645827492d889d57c9f3f475fc"
format_stamp: "Formatted at 2026-09-25 22:41:13 on dist-test-slave-csnl"
server_key: "5f95d28a3afe2b9c1f8e5d13753c85ea"
server_key_iv: "c6fc485c35d2d41098b975f3e6a710eb"
server_key_version: "encryptionkey@0"
I20260925 22:41:13.077275 7165 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:41:13.080132 7179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:13.081702 7165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:41:13.081900 7165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "6bde3f645827492d889d57c9f3f475fc"
format_stamp: "Formatted at 2026-09-25 22:41:13 on dist-test-slave-csnl"
server_key: "5f95d28a3afe2b9c1f8e5d13753c85ea"
server_key_iv: "c6fc485c35d2d41098b975f3e6a710eb"
server_key_version: "encryptionkey@0"
I20260925 22:41:13.082052 7165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:13.087277 7165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:13.088007 7165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:13.088199 7165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:13.088778 7165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:13.089725 7165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:13.089797 7165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:13.089888 7165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:13.089939 7165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:13.099588 7165 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:45131
I20260925 22:41:13.099656 7292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:45131 every 8 connection(s)
I20260925 22:41:13.100669 7165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:41:13.107270 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 7165
I20260925 22:41:13.107402 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance
I20260925 22:41:13.107487 7013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:13.102722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:36865 (local address 127.2.74.254:43731)
0925 22:41:13.102875 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:13.102880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:13.104218 (+ 1338us) server_negotiation.cc:407] Connection header received
0925 22:41:13.105620 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:13.105624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:13.105679 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:13.105764 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:13.106940 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:13.106944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:13.106947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:41:13.106961 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:13.107008 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:41:13.107022 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:13.107139 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:13.107261 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 22:41:13.107311 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 22:41:13.107754 2347 external_mini_cluster.cc:1428] Setting key 75bff8a010d401b635a477395f16afc0
I20260925 22:41:13.108979 7295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:13.103295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:43731 (local address 127.2.74.194:36865)
0925 22:41:13.104032 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:13.104099 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:13.105361 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:13.105953 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:13.105960 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:13.106061 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:13.106624 (+ 563us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:41:13.106626 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:13.106671 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:41:13.106748 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:13.107112 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:13.107118 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:13.107122 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 22:41:13.107267 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 22:41:13.107596 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":124,"threads_started":1}
I20260925 22:41:13.110046 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:0
--local_ip_for_outbound_sockets=127.2.74.195
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:43731
--builtin_ntp_servers=127.2.74.212:45611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 22:41:13.110363 7293 heartbeater.cc:347] Connected to a master server at 127.2.74.254:43731
I20260925 22:41:13.110647 7293 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:13.111208 7293 heartbeater.cc:511] Master 127.2.74.254:43731 requested a full tablet report, sending...
I20260925 22:41:13.112560 6974 ts_manager.cc:194] Registered new tserver with Master: 6bde3f645827492d889d57c9f3f475fc (127.2.74.194:45131)
W20260925 22:41:13.226092 7296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:13.226394 7296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:13.226437 7296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:13.230612 7296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:13.230695 7296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:13.230804 7296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:41:13.236119 7296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:45611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:43731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:13.237299 7296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:13.238832 7296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:13.248773 7302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:13.248916 7296 server_base.cc:1034] running on GCE node
W20260925 22:41:13.248936 7304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:13.248816 7301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:13.249527 7296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:13.250185 7296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:13.251361 7296 hybrid_clock.cc:648] HybridClock initialized: now 1790376073251328 us; error 50 us; skew 500 ppm
I20260925 22:41:13.253209 7296 webserver.cc:460] Webserver started at http://127.2.74.195:44951/ using document root <none> and password file <none>
I20260925 22:41:13.253813 7296 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:13.253907 7296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:13.254146 7296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:13.256301 7296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-2/data/instance:
uuid: "a8ac7f82674a43f9a1e1fac49fc70906"
format_stamp: "Formatted at 2026-09-25 22:41:13 on dist-test-slave-csnl"
server_key: "5fb74b4221ff990636ff84a61a8ac4e6"
server_key_iv: "bd162610131acc41d448df773bcc71a8"
server_key_version: "encryptionkey@0"
I20260925 22:41:13.256901 7296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance:
uuid: "a8ac7f82674a43f9a1e1fac49fc70906"
format_stamp: "Formatted at 2026-09-25 22:41:13 on dist-test-slave-csnl"
server_key: "5fb74b4221ff990636ff84a61a8ac4e6"
server_key_iv: "bd162610131acc41d448df773bcc71a8"
server_key_version: "encryptionkey@0"
I20260925 22:41:13.261013 7296 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:41:13.263660 7310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:13.264976 7296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:41:13.265117 7296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "a8ac7f82674a43f9a1e1fac49fc70906"
format_stamp: "Formatted at 2026-09-25 22:41:13 on dist-test-slave-csnl"
server_key: "5fb74b4221ff990636ff84a61a8ac4e6"
server_key_iv: "bd162610131acc41d448df773bcc71a8"
server_key_version: "encryptionkey@0"
I20260925 22:41:13.265242 7296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:13.269964 7296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:13.270601 7296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:13.270783 7296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:13.271407 7296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:13.272322 7296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:13.272389 7296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:13.272468 7296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:13.272518 7296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:13.281574 7296 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:41053
I20260925 22:41:13.281602 7423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:41053 every 8 connection(s)
I20260925 22:41:13.282568 7296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:41:13.287580 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 7296
I20260925 22:41:13.287715 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance
I20260925 22:41:13.288021 2347 external_mini_cluster.cc:1428] Setting key 759d61680bd5b32c1cd5ae8c30a0eecc
I20260925 22:41:13.289507 7013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:13.284616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:35029 (local address 127.2.74.254:43731)
0925 22:41:13.284762 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:13.284766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:13.286050 (+ 1284us) server_negotiation.cc:407] Connection header received
0925 22:41:13.287579 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:13.287582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:13.287630 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:13.287722 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:13.289010 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:13.289014 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:13.289017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:41:13.289031 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:13.289059 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:41:13.289068 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:13.289163 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:13.289322 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 22:41:13.289369 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:41:13.290568 7426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:13.285080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:43731 (local address 127.2.74.195:35029)
0925 22:41:13.285873 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:13.285930 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:13.287334 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:13.287929 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:13.287937 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:13.288035 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:13.288724 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:41:13.288726 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:13.288769 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:41:13.288846 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:13.289163 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:13.289174 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:13.289183 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 22:41:13.289371 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 22:41:13.289656 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":163,"threads_started":1}
I20260925 22:41:13.291682 7424 heartbeater.cc:347] Connected to a master server at 127.2.74.254:43731
I20260925 22:41:13.291913 7424 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:13.292299 7424 heartbeater.cc:511] Master 127.2.74.254:43731 requested a full tablet report, sending...
I20260925 22:41:13.293256 6974 ts_manager.cc:194] Registered new tserver with Master: a8ac7f82674a43f9a1e1fac49fc70906 (127.2.74.195:41053)
I20260925 22:41:13.302397 2347 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:41:13.306166 7013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:13.304443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35706 (local address 127.2.74.254:43731)
0925 22:41:13.304613 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:13.304618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:13.304736 (+ 118us) server_negotiation.cc:407] Connection header received
0925 22:41:13.304915 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:13.304918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:13.304962 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:13.305050 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:13.305726 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:13.305729 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:13.305732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:41:13.305744 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:13.305766 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:41:13.305782 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:13.305877 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:13.305971 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 22:41:13.306009 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 22:41:13.309291 6974 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:41:13.312548 6974 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:41:13.324887 7436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:13.321689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:41053 (local address 127.0.0.1:56754)
0925 22:41:13.322202 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:13.322246 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:13.322338 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:13.323179 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:13.323188 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:13.323274 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:13.323904 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:41:13.323906 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:13.323928 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:41:13.324007 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:13.324508 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:13.324517 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:13.324523 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:41:13.324633 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 22:41:13.324708 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":75,"threads_started":1}
I20260925 22:41:13.324970 7437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:13.321830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56754 (local address 127.2.74.195:41053)
0925 22:41:13.322632 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:13.322646 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:13.322672 (+ 26us) server_negotiation.cc:407] Connection header received
0925 22:41:13.322732 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:13.322740 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:13.323001 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:13.323588 (+ 587us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:13.324332 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:13.324337 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:13.324347 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:41:13.324354 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:13.324384 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:41:13.324399 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:13.324522 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:13.324673 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 22:41:13.324762 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":114,"threads_started":1}
I20260925 22:41:13.326164 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:13.321797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:39735 (local address 127.0.0.1:33002)
0925 22:41:13.322583 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:13.322598 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:13.322672 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:13.324597 (+ 1925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:13.324602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:13.324621 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:13.325283 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:41:13.325285 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:13.325294 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:41:13.325319 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:13.325896 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:13.325900 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:13.325903 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 22:41:13.325985 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 22:41:13.326024 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":102,"threads_started":1}
I20260925 22:41:13.326334 7438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:13.322473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:45131 (local address 127.0.0.1:38160)
0925 22:41:13.323048 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:13.323063 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:13.323138 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:13.324646 (+ 1508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:13.324649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:13.324665 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:13.325681 (+ 1016us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:41:13.325683 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:13.325695 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:41:13.325721 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:13.326107 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:13.326110 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:13.326113 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 22:41:13.326183 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 22:41:13.326217 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":92,"threads_started":1}
I20260925 22:41:13.326769 7440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:13.322782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33002 (local address 127.2.74.193:39735)
0925 22:41:13.323608 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:13.323620 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:13.323642 (+ 22us) server_negotiation.cc:407] Connection header received
0925 22:41:13.323693 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:13.323700 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:13.323870 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:13.324009 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:13.325495 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:13.325501 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:13.325514 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:41:13.325527 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:13.325570 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:41:13.325607 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:13.325715 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:13.326199 (+ 484us) server_negotiation.cc:299] Negotiation successful
0925 22:41:13.326639 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":123,"threads_started":1}
I20260925 22:41:13.326799 7439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:13.322666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38160 (local address 127.2.74.194:45131)
0925 22:41:13.323254 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:13.323266 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:13.323290 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:41:13.323341 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:13.323346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:13.323862 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:13.324009 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:13.325827 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:13.325833 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:13.325845 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:41:13.325859 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:13.325900 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:41:13.325945 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:13.326064 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:13.326563 (+ 499us) server_negotiation.cc:299] Negotiation successful
0925 22:41:13.326638 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":95,"threads_started":1}
I20260925 22:41:13.328665 7358 tablet_service.cc:1467] Processing CreateTablet for tablet c3cb744b29da4906a9f812fdf9ecb692 (DEFAULT_TABLE table=test-table [id=f6859afb14ee441fb16c3d594ec58f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:13.329839 7358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3cb744b29da4906a9f812fdf9ecb692. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:13.329540 7096 tablet_service.cc:1467] Processing CreateTablet for tablet c3cb744b29da4906a9f812fdf9ecb692 (DEFAULT_TABLE table=test-table [id=f6859afb14ee441fb16c3d594ec58f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:13.329558 7227 tablet_service.cc:1467] Processing CreateTablet for tablet c3cb744b29da4906a9f812fdf9ecb692 (DEFAULT_TABLE table=test-table [id=f6859afb14ee441fb16c3d594ec58f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:13.330510 7227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3cb744b29da4906a9f812fdf9ecb692. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:13.330507 7096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3cb744b29da4906a9f812fdf9ecb692. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:13.336997 7441 tablet_bootstrap.cc:492] T c3cb744b29da4906a9f812fdf9ecb692 P c9ac3ee73cc445fca965f090d050f4f9: Bootstrap starting.
I20260925 22:41:13.338184 7442 tablet_bootstrap.cc:492] T c3cb744b29da4906a9f812fdf9ecb692 P 6bde3f645827492d889d57c9f3f475fc: Bootstrap starting.
I20260925 22:41:13.339318 7443 tablet_bootstrap.cc:492] T c3cb744b29da4906a9f812fdf9ecb692 P a8ac7f82674a43f9a1e1fac49fc70906: Bootstrap starting.
I20260925 22:41:13.340055 7441 tablet_bootstrap.cc:654] T c3cb744b29da4906a9f812fdf9ecb692 P c9ac3ee73cc445fca965f090d050f4f9: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:13.340312 7442 tablet_bootstrap.cc:654] T c3cb744b29da4906a9f812fdf9ecb692 P 6bde3f645827492d889d57c9f3f475fc: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:13.340873 7441 log.cc:824] T c3cb744b29da4906a9f812fdf9ecb692 P c9ac3ee73cc445fca965f090d050f4f9: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:13.341104 7442 log.cc:824] T c3cb744b29da4906a9f812fdf9ecb692 P 6bde3f645827492d889d57c9f3f475fc: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:13.341456 7443 tablet_bootstrap.cc:654] T c3cb744b29da4906a9f812fdf9ecb692 P a8ac7f82674a43f9a1e1fac49fc70906: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:13.342216 7443 log.cc:824] T c3cb744b29da4906a9f812fdf9ecb692 P a8ac7f82674a43f9a1e1fac49fc70906: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:13.343092 7442 tablet_bootstrap.cc:492] T c3cb744b29da4906a9f812fdf9ecb692 P 6bde3f645827492d889d57c9f3f475fc: No bootstrap required, opened a new log
I20260925 22:41:13.343092 7441 tablet_bootstrap.cc:492] T c3cb744b29da4906a9f812fdf9ecb692 P c9ac3ee73cc445fca965f090d050f4f9: No bootstrap required, opened a new log
I20260925 22:41:13.343293 7442 ts_tablet_manager.cc:1397] T c3cb744b29da4906a9f812fdf9ecb692 P 6bde3f645827492d889d57c9f3f475fc: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:41:13.343293 7441 ts_tablet_manager.cc:1397] T c3cb744b29da4906a9f812fdf9ecb692 P c9ac3ee73cc445fca965f090d050f4f9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:41:13.344004 7443 tablet_bootstrap.cc:492] T c3cb744b29da4906a9f812fdf9ecb692 P a8ac7f82674a43f9a1e1fac49fc70906: No bootstrap required, opened a new log
I20260925 22:41:13.344203 7443 ts_tablet_manager.cc:1397] T c3cb744b29da4906a9f812fdf9ecb692 P a8ac7f82674a43f9a1e1fac49fc70906: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 22:41:13.347545 7443 raft_consensus.cc:348] T c3cb744b29da4906a9f812fdf9ecb692 P a8ac7f82674a43f9a1e1fac49fc70906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ac7f82674a43f9a1e1fac49fc70906" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 41053 } } peers { permanent_uuid: "c9ac3ee73cc445fca965f090d050f4f9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 39735 } } peers { permanent_uuid: "6bde3f645827492d889d57c9f3f475fc" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 45131 } }
I20260925 22:41:13.347864 7443 raft_consensus.cc:374] T c3cb744b29da4906a9f812fdf9ecb692 P a8ac7f82674a43f9a1e1fac49fc70906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:13.347929 7443 raft_consensus.cc:729] T c3cb744b29da4906a9f812fdf9ecb692 P a8ac7f82674a43f9a1e1fac49fc70906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8ac7f82674a43f9a1e1fac49fc70906, State: Initialized, Role: FOLLOWER
I20260925 22:41:13.348024 7441 raft_consensus.cc:348] T c3cb744b29da4906a9f812fdf9ecb692 P c9ac3ee73cc445fca965f090d050f4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ac7f82674a43f9a1e1fac49fc70906" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 41053 } } peers { permanent_uuid: "c9ac3ee73cc445fca965f090d050f4f9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 39735 } } peers { permanent_uuid: "6bde3f645827492d889d57c9f3f475fc" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 45131 } }
I20260925 22:41:13.348024 7442 raft_consensus.cc:348] T c3cb744b29da4906a9f812fdf9ecb692 P 6bde3f645827492d889d57c9f3f475fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ac7f82674a43f9a1e1fac49fc70906" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 41053 } } peers { permanent_uuid: "c9ac3ee73cc445fca965f090d050f4f9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 39735 } } peers { permanent_uuid: "6bde3f645827492d889d57c9f3f475fc" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 45131 } }
I20260925 22:41:13.348266 7441 raft_consensus.cc:374] T c3cb744b29da4906a9f812fdf9ecb692 P c9ac3ee73cc445fca965f090d050f4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:13.348266 7442 raft_consensus.cc:374] T c3cb744b29da4906a9f812fdf9ecb692 P 6bde3f645827492d889d57c9f3f475fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:13.348317 7442 raft_consensus.cc:729] T c3cb744b29da4906a9f812fdf9ecb692 P 6bde3f645827492d889d57c9f3f475fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bde3f645827492d889d57c9f3f475fc, State: Initialized, Role: FOLLOWER
I20260925 22:41:13.348317 7441 raft_consensus.cc:729] T c3cb744b29da4906a9f812fdf9ecb692 P c9ac3ee73cc445fca965f090d050f4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9ac3ee73cc445fca965f090d050f4f9, State: Initialized, Role: FOLLOWER
I20260925 22:41:13.348443 7443 consensus_queue.cc:260] T c3cb744b29da4906a9f812fdf9ecb692 P a8ac7f82674a43f9a1e1fac49fc70906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ac7f82674a43f9a1e1fac49fc70906" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 41053 } } peers { permanent_uuid: "c9ac3ee73cc445fca965f090d050f4f9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 39735 } } peers { permanent_uuid: "6bde3f645827492d889d57c9f3f475fc" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 45131 } }
I20260925 22:41:13.348801 7441 consensus_queue.cc:260] T c3cb744b29da4906a9f812fdf9ecb692 P c9ac3ee73cc445fca965f090d050f4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ac7f82674a43f9a1e1fac49fc70906" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 41053 } } peers { permanent_uuid: "c9ac3ee73cc445fca965f090d050f4f9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 39735 } } peers { permanent_uuid: "6bde3f645827492d889d57c9f3f475fc" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 45131 } }
I20260925 22:41:13.348802 7442 consensus_queue.cc:260] T c3cb744b29da4906a9f812fdf9ecb692 P 6bde3f645827492d889d57c9f3f475fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ac7f82674a43f9a1e1fac49fc70906" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 41053 } } peers { permanent_uuid: "c9ac3ee73cc445fca965f090d050f4f9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 39735 } } peers { permanent_uuid: "6bde3f645827492d889d57c9f3f475fc" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 45131 } }
I20260925 22:41:13.349354 7424 heartbeater.cc:503] Master 127.2.74.254:43731 was elected leader, sending a full tablet report...
I20260925 22:41:13.349553 7443 ts_tablet_manager.cc:1428] T c3cb744b29da4906a9f812fdf9ecb692 P a8ac7f82674a43f9a1e1fac49fc70906: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260925 22:41:13.349817 7441 ts_tablet_manager.cc:1428] T c3cb744b29da4906a9f812fdf9ecb692 P c9ac3ee73cc445fca965f090d050f4f9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:41:13.349936 7293 heartbeater.cc:503] Master 127.2.74.254:43731 was elected leader, sending a full tablet report...
I20260925 22:41:13.350448 7162 heartbeater.cc:503] Master 127.2.74.254:43731 was elected leader, sending a full tablet report...
I20260925 22:41:13.351366 7442 ts_tablet_manager.cc:1428] T c3cb744b29da4906a9f812fdf9ecb692 P 6bde3f645827492d889d57c9f3f475fc: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
W20260925 22:41:13.351709 7294 tablet.cc:2369] T c3cb744b29da4906a9f812fdf9ecb692 P 6bde3f645827492d889d57c9f3f475fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:41:13.418344 7163 tablet.cc:2369] T c3cb744b29da4906a9f812fdf9ecb692 P c9ac3ee73cc445fca965f090d050f4f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:41:13.533929 7425 tablet.cc:2369] T c3cb744b29da4906a9f812fdf9ecb692 P a8ac7f82674a43f9a1e1fac49fc70906: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:41:13.680678 7449 raft_consensus.cc:482] T c3cb744b29da4906a9f812fdf9ecb692 P c9ac3ee73cc445fca965f090d050f4f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:41:13.680881 7449 raft_consensus.cc:504] T c3cb744b29da4906a9f812fdf9ecb692 P c9ac3ee73cc445fca965f090d050f4f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ac7f82674a43f9a1e1fac49fc70906" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 41053 } } peers { permanent_uuid: "c9ac3ee73cc445fca965f090d050f4f9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 39735 } } peers { permanent_uuid: "6bde3f645827492d889d57c9f3f475fc" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 45131 } }
I20260925 22:41:13.681965 7449 leader_election.cc:288] T c3cb744b29da4906a9f812fdf9ecb692 P c9ac3ee73cc445fca965f090d050f4f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8ac7f82674a43f9a1e1fac49fc70906 (127.2.74.195:41053), 6bde3f645827492d889d57c9f3f475fc (127.2.74.194:45131)
I20260925 22:41:13.684376 7437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:13.682374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:59935 (local address 127.2.74.195:41053)
0925 22:41:13.682521 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:13.682525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:13.682658 (+ 133us) server_negotiation.cc:407] Connection header received
0925 22:41:13.682882 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:13.682886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:13.682929 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:13.683011 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:13.683901 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:13.683905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:13.683908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:41:13.683926 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:13.683949 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:41:13.683959 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:13.684052 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:13.684202 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 22:41:13.684243 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 22:41:13.684392 7450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:13.682297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:41053 (local address 127.2.74.193:59935)
0925 22:41:13.682573 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:13.682590 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:13.682732 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:13.683144 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:13.683149 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:13.683168 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:13.683712 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:41:13.683715 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:13.683729 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:41:13.683766 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:13.684061 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:13.684066 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:13.684073 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 22:41:13.684193 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 22:41:13.684233 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":101,"threads_started":1}
I20260925 22:41:13.685195 7439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:13.682998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:55163 (local address 127.2.74.194:45131)
0925 22:41:13.683136 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:13.683140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:13.683691 (+ 551us) server_negotiation.cc:407] Connection header received
0925 22:41:13.683745 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:13.683749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:13.683792 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:13.683871 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:13.684763 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:13.684766 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:13.684769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:41:13.684782 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:13.684804 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:41:13.684817 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:13.684888 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:13.685045 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 22:41:13.685083 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:41:13.685211 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:13.682692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:45131 (local address 127.2.74.193:55163)
0925 22:41:13.683258 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:13.683273 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:13.683385 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:13.683907 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:13.683910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:13.683927 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:13.684572 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:41:13.684574 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:13.684583 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:41:13.684610 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:13.684947 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:13.684949 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:13.684951 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 22:41:13.685039 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 22:41:13.685074 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":66,"threads_started":1}
I20260925 22:41:13.685145 7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3cb744b29da4906a9f812fdf9ecb692" candidate_uuid: "c9ac3ee73cc445fca965f090d050f4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8ac7f82674a43f9a1e1fac49fc70906" is_pre_election: true
I20260925 22:41:13.685473 7378 raft_consensus.cc:2454] T c3cb744b29da4906a9f812fdf9ecb692 P a8ac7f82674a43f9a1e1fac49fc70906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9ac3ee73cc445fca965f090d050f4f9 in term 0.
I20260925 22:41:13.685817 7247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3cb744b29da4906a9f812fdf9ecb692" candidate_uuid: "c9ac3ee73cc445fca965f090d050f4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bde3f645827492d889d57c9f3f475fc" is_pre_election: true
I20260925 22:41:13.686021 7051 leader_election.cc:302] T c3cb744b29da4906a9f812fdf9ecb692 P c9ac3ee73cc445fca965f090d050f4f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8ac7f82674a43f9a1e1fac49fc70906, c9ac3ee73cc445fca965f090d050f4f9; no voters:
I20260925 22:41:13.686091 7247 raft_consensus.cc:2454] T c3cb744b29da4906a9f812fdf9ecb692 P 6bde3f645827492d889d57c9f3f475fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9ac3ee73cc445fca965f090d050f4f9 in term 0.
I20260925 22:41:13.686316 7449 raft_consensus.cc:2781] T c3cb744b29da4906a9f812fdf9ecb692 P c9ac3ee73cc445fca965f090d050f4f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:41:13.686386 7449 raft_consensus.cc:482] T c3cb744b29da4906a9f812fdf9ecb692 P c9ac3ee73cc445fca965f090d050f4f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:41:13.686424 7449 raft_consensus.cc:3037] T c3cb744b29da4906a9f812fdf9ecb692 P c9ac3ee73cc445fca965f090d050f4f9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:13.687800 7449 raft_consensus.cc:504] T c3cb744b29da4906a9f812fdf9ecb692 P c9ac3ee73cc445fca965f090d050f4f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ac7f82674a43f9a1e1fac49fc70906" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 41053 } } peers { permanent_uuid: "c9ac3ee73cc445fca965f090d050f4f9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 39735 } } peers { permanent_uuid: "6bde3f645827492d889d57c9f3f475fc" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 45131 } }
I20260925 22:41:13.688205 7449 leader_election.cc:288] T c3cb744b29da4906a9f812fdf9ecb692 P c9ac3ee73cc445fca965f090d050f4f9 [CANDIDATE]: Term 1 election: Requested vote from peers a8ac7f82674a43f9a1e1fac49fc70906 (127.2.74.195:41053), 6bde3f645827492d889d57c9f3f475fc (127.2.74.194:45131)
I20260925 22:41:13.688663 7247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3cb744b29da4906a9f812fdf9ecb692" candidate_uuid: "c9ac3ee73cc445fca965f090d050f4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bde3f645827492d889d57c9f3f475fc"
I20260925 22:41:13.688800 7247 raft_consensus.cc:3037] T c3cb744b29da4906a9f812fdf9ecb692 P 6bde3f645827492d889d57c9f3f475fc [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:13.690146 7247 raft_consensus.cc:2454] T c3cb744b29da4906a9f812fdf9ecb692 P 6bde3f645827492d889d57c9f3f475fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9ac3ee73cc445fca965f090d050f4f9 in term 1.
I20260925 22:41:13.690452 7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3cb744b29da4906a9f812fdf9ecb692" candidate_uuid: "c9ac3ee73cc445fca965f090d050f4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8ac7f82674a43f9a1e1fac49fc70906"
I20260925 22:41:13.690514 7051 leader_election.cc:302] T c3cb744b29da4906a9f812fdf9ecb692 P c9ac3ee73cc445fca965f090d050f4f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bde3f645827492d889d57c9f3f475fc, c9ac3ee73cc445fca965f090d050f4f9; no voters:
I20260925 22:41:13.690601 7378 raft_consensus.cc:3037] T c3cb744b29da4906a9f812fdf9ecb692 P a8ac7f82674a43f9a1e1fac49fc70906 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:13.690762 7449 raft_consensus.cc:2781] T c3cb744b29da4906a9f812fdf9ecb692 P c9ac3ee73cc445fca965f090d050f4f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:41:13.691044 7449 raft_consensus.cc:686] T c3cb744b29da4906a9f812fdf9ecb692 P c9ac3ee73cc445fca965f090d050f4f9 [term 1 LEADER]: Becoming Leader. State: Replica: c9ac3ee73cc445fca965f090d050f4f9, State: Running, Role: LEADER
I20260925 22:41:13.691418 7449 consensus_queue.cc:237] T c3cb744b29da4906a9f812fdf9ecb692 P c9ac3ee73cc445fca965f090d050f4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ac7f82674a43f9a1e1fac49fc70906" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 41053 } } peers { permanent_uuid: "c9ac3ee73cc445fca965f090d050f4f9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 39735 } } peers { permanent_uuid: "6bde3f645827492d889d57c9f3f475fc" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 45131 } }
I20260925 22:41:13.691895 7378 raft_consensus.cc:2454] T c3cb744b29da4906a9f812fdf9ecb692 P a8ac7f82674a43f9a1e1fac49fc70906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9ac3ee73cc445fca965f090d050f4f9 in term 1.
I20260925 22:41:13.694595 6972 catalog_manager.cc:5495] T c3cb744b29da4906a9f812fdf9ecb692 P c9ac3ee73cc445fca965f090d050f4f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9ac3ee73cc445fca965f090d050f4f9 (127.2.74.193). New cstate: current_term: 1 leader_uuid: "c9ac3ee73cc445fca965f090d050f4f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ac7f82674a43f9a1e1fac49fc70906" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 41053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9ac3ee73cc445fca965f090d050f4f9" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 39735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bde3f645827492d889d57c9f3f475fc" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 45131 } health_report { overall_health: UNKNOWN } } }
I20260925 22:41:13.782212 7440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:13.780780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33004 (local address 127.2.74.193:39735)
0925 22:41:13.780914 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:13.780918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:13.780938 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:41:13.781040 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:13.781043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:13.781103 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:13.781171 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:13.781822 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:13.781825 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:13.781827 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:41:13.781838 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:13.781856 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:41:13.781864 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:13.781957 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:13.781990 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 22:41:13.782023 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:41:13.788720 7378 raft_consensus.cc:1261] T c3cb744b29da4906a9f812fdf9ecb692 P a8ac7f82674a43f9a1e1fac49fc70906 [term 1 FOLLOWER]: Refusing update from remote peer c9ac3ee73cc445fca965f090d050f4f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:41:13.788719 7247 raft_consensus.cc:1261] T c3cb744b29da4906a9f812fdf9ecb692 P 6bde3f645827492d889d57c9f3f475fc [term 1 FOLLOWER]: Refusing update from remote peer c9ac3ee73cc445fca965f090d050f4f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:41:13.789479 7449 consensus_queue.cc:1035] T c3cb744b29da4906a9f812fdf9ecb692 P c9ac3ee73cc445fca965f090d050f4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8ac7f82674a43f9a1e1fac49fc70906" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 41053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:41:13.789712 7452 consensus_queue.cc:1035] T c3cb744b29da4906a9f812fdf9ecb692 P c9ac3ee73cc445fca965f090d050f4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bde3f645827492d889d57c9f3f475fc" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 45131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:41:13.796767 7455 mvcc.cc:205] Tried to move back new op lower bound from 7333380398230294528 to 7333380397843165184. Current Snapshot: MvccSnapshot[applied={T|T < 7333380398230294528}]
I20260925 22:41:13.807475 6973 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:41:13.807682 6973 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:41:13.810590 6973 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a83abb7e9d04896b926926010828cfe: Sending DeleteTablet for 3 replicas of tablet c3cb744b29da4906a9f812fdf9ecb692
I20260925 22:41:13.811574 7227 tablet_service.cc:1514] Processing DeleteTablet for tablet c3cb744b29da4906a9f812fdf9ecb692 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:13 UTC) from {username='slave'} at 127.0.0.1:38160
I20260925 22:41:13.811575 7096 tablet_service.cc:1514] Processing DeleteTablet for tablet c3cb744b29da4906a9f812fdf9ecb692 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:13 UTC) from {username='slave'} at 127.0.0.1:33002
I20260925 22:41:13.812172 7466 tablet_replica.cc:330] T c3cb744b29da4906a9f812fdf9ecb692 P 6bde3f645827492d889d57c9f3f475fc: stopping tablet replica
I20260925 22:41:13.812155 7358 tablet_service.cc:1514] Processing DeleteTablet for tablet c3cb744b29da4906a9f812fdf9ecb692 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:13 UTC) from {username='slave'} at 127.0.0.1:56754
I20260925 22:41:13.812579 7466 raft_consensus.cc:2229] T c3cb744b29da4906a9f812fdf9ecb692 P 6bde3f645827492d889d57c9f3f475fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:41:13.812633 7467 tablet_replica.cc:330] T c3cb744b29da4906a9f812fdf9ecb692 P a8ac7f82674a43f9a1e1fac49fc70906: stopping tablet replica
I20260925 22:41:13.812969 7466 raft_consensus.cc:2258] T c3cb744b29da4906a9f812fdf9ecb692 P 6bde3f645827492d889d57c9f3f475fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:41:13.813119 7467 raft_consensus.cc:2229] T c3cb744b29da4906a9f812fdf9ecb692 P a8ac7f82674a43f9a1e1fac49fc70906 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:41:13.813380 7465 tablet_replica.cc:330] T c3cb744b29da4906a9f812fdf9ecb692 P c9ac3ee73cc445fca965f090d050f4f9: stopping tablet replica
I20260925 22:41:13.813556 7467 raft_consensus.cc:2258] T c3cb744b29da4906a9f812fdf9ecb692 P a8ac7f82674a43f9a1e1fac49fc70906 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:41:13.813652 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 7034
I20260925 22:41:13.813691 7465 raft_consensus.cc:2229] T c3cb744b29da4906a9f812fdf9ecb692 P c9ac3ee73cc445fca965f090d050f4f9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:41:13.816129 7467 ts_tablet_manager.cc:1905] T c3cb744b29da4906a9f812fdf9ecb692 P a8ac7f82674a43f9a1e1fac49fc70906: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:41:13.816972 7466 ts_tablet_manager.cc:1905] T c3cb744b29da4906a9f812fdf9ecb692 P 6bde3f645827492d889d57c9f3f475fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:41:13.819176 7467 ts_tablet_manager.cc:1918] T c3cb744b29da4906a9f812fdf9ecb692 P a8ac7f82674a43f9a1e1fac49fc70906: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:41:13.819293 7467 log.cc:1196] T c3cb744b29da4906a9f812fdf9ecb692 P a8ac7f82674a43f9a1e1fac49fc70906: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-2/wal/wals/c3cb744b29da4906a9f812fdf9ecb692
I20260925 22:41:13.819619 7467 ts_tablet_manager.cc:1939] T c3cb744b29da4906a9f812fdf9ecb692 P a8ac7f82674a43f9a1e1fac49fc70906: Deleting consensus metadata
I20260925 22:41:13.819969 7466 ts_tablet_manager.cc:1918] T c3cb744b29da4906a9f812fdf9ecb692 P 6bde3f645827492d889d57c9f3f475fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:41:13.820070 7466 log.cc:1196] T c3cb744b29da4906a9f812fdf9ecb692 P 6bde3f645827492d889d57c9f3f475fc: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376047580784-2347-0/minicluster-data/ts-1/wal/wals/c3cb744b29da4906a9f812fdf9ecb692
I20260925 22:41:13.820379 7466 ts_tablet_manager.cc:1939] T c3cb744b29da4906a9f812fdf9ecb692 P 6bde3f645827492d889d57c9f3f475fc: Deleting consensus metadata
I20260925 22:41:13.821183 6960 catalog_manager.cc:4841] TS a8ac7f82674a43f9a1e1fac49fc70906 (127.2.74.195:41053): tablet c3cb744b29da4906a9f812fdf9ecb692 (table test-table [id=f6859afb14ee441fb16c3d594ec58f0c]) successfully deleted
I20260925 22:41:13.821518 6960 catalog_manager.cc:4841] TS 6bde3f645827492d889d57c9f3f475fc (127.2.74.194:45131): tablet c3cb744b29da4906a9f812fdf9ecb692 (table test-table [id=f6859afb14ee441fb16c3d594ec58f0c]) successfully deleted
W20260925 22:41:13.823699 6959 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.74.193:39735 (error 108)
I20260925 22:41:13.824473 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 7165
I20260925 22:41:13.824788 7438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:13.824426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:39735 (local address 127.0.0.1:33016)
0925 22:41:13.824596 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:13.824679 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.193:39735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 22:41:13.825209 6959 catalog_manager.cc:4584] TS c9ac3ee73cc445fca965f090d050f4f9 (127.2.74.193:39735): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c3cb744b29da4906a9f812fdf9ecb692: Network error: Client connection negotiation failed: client connection to 127.2.74.193:39735: connect: Connection refused (error 111)
I20260925 22:41:13.830345 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 7296
I20260925 22:41:13.836109 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 6943
2026-09-25T22:41:13Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1361 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 22:41:13.857843 2347 mini_oidc.cc:152] Starting JWKS server
I20260925 22:41:13.859594 2347 webserver.cc:460] Webserver started at https://127.0.0.1:38695/ using document root <none> and password file <none>
I20260925 22:41:13.859800 2347 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 22:41:13.860103 2347 webserver.cc:460] Webserver started at http://127.0.0.1:45945/ using document root <none> and password file <none>
2026-09-25T22:41:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:41:13Z Disabled control of system clock
I20260925 22:41:13.869844 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:39123
--webserver_interface=127.2.74.254
--webserver_port=0
--builtin_ntp_servers=127.2.74.212:39315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:39123
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38695/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376047580784-2347-0/testchainca.cert with env {}
W20260925 22:41:13.983637 7474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:13.983950 7474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:13.983992 7474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:13.988382 7474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:41:13.988472 7474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:13.988489 7474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:13.988510 7474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:41:13.988526 7474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 22:41:13.988543 7474 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 22:41:13.988556 7474 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38695/jwks.json
I20260925 22:41:13.994207 7474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:39315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:39123
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:39123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38695/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376047580784-2347-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:13.995467 7474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:13.996480 7474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:14.005184 7482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:14.005393 7474 server_base.cc:1034] running on GCE node
W20260925 22:41:14.005184 7480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:14.005297 7479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:14.006319 7474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:14.007630 7474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:14.008841 7474 hybrid_clock.cc:648] HybridClock initialized: now 1790376074008836 us; error 39 us; skew 500 ppm
I20260925 22:41:14.011072 7474 webserver.cc:460] Webserver started at http://127.2.74.254:33147/ using document root <none> and password file <none>
I20260925 22:41:14.011727 7474 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:14.011837 7474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:14.012085 7474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:14.014324 7474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "2da75716d0dc4df6a98b83e0c6e68fe7"
format_stamp: "Formatted at 2026-09-25 22:41:14 on dist-test-slave-csnl"
server_key: "a674b51f04c5992cde4c2cd13702dfc8"
server_key_iv: "7a989f98159a5d7cd0551c850f85983e"
server_key_version: "encryptionkey@0"
I20260925 22:41:14.015038 7474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "2da75716d0dc4df6a98b83e0c6e68fe7"
format_stamp: "Formatted at 2026-09-25 22:41:14 on dist-test-slave-csnl"
server_key: "a674b51f04c5992cde4c2cd13702dfc8"
server_key_iv: "7a989f98159a5d7cd0551c850f85983e"
server_key_version: "encryptionkey@0"
I20260925 22:41:14.019320 7474 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:41:14.022511 7488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:14.024197 7474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:41:14.024435 7474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "2da75716d0dc4df6a98b83e0c6e68fe7"
format_stamp: "Formatted at 2026-09-25 22:41:14 on dist-test-slave-csnl"
server_key: "a674b51f04c5992cde4c2cd13702dfc8"
server_key_iv: "7a989f98159a5d7cd0551c850f85983e"
server_key_version: "encryptionkey@0"
I20260925 22:41:14.024596 7474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:14.058579 7474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:14.059420 7474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:14.059659 7474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:14.067206 7474 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:39123
I20260925 22:41:14.067243 7540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:39123 every 8 connection(s)
I20260925 22:41:14.068287 7474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:41:14.071789 7541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:14.076828 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 7474
I20260925 22:41:14.076982 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:41:14.077275 2347 external_mini_cluster.cc:1428] Setting key 8c5e9f352eefb306f46606fb1d28f5e2
I20260925 22:41:14.078254 7541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2da75716d0dc4df6a98b83e0c6e68fe7: Bootstrap starting.
I20260925 22:41:14.081305 7541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2da75716d0dc4df6a98b83e0c6e68fe7: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:14.082261 7541 log.cc:824] T 00000000000000000000000000000000 P 2da75716d0dc4df6a98b83e0c6e68fe7: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:14.084690 7541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2da75716d0dc4df6a98b83e0c6e68fe7: No bootstrap required, opened a new log
I20260925 22:41:14.088843 7544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:14.078833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50720 (local address 127.2.74.254:39123)
0925 22:41:14.079673 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:14.079696 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:14.079748 (+ 52us) server_negotiation.cc:407] Connection header received
0925 22:41:14.080640 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:14.080666 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:14.081253 (+ 587us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:14.081683 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:14.082178 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:14.083589 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:14.084613 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:14.085329 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:14.085496 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:14.086363 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:14.086369 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:14.086388 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:41:14.086943 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:14.087060 (+ 117us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:41:14.087090 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:14.087190 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:14.087373 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 22:41:14.087679 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":141,"threads_started":1}
I20260925 22:41:14.089381 7541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2da75716d0dc4df6a98b83e0c6e68fe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da75716d0dc4df6a98b83e0c6e68fe7" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 39123 } }
I20260925 22:41:14.089610 7541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2da75716d0dc4df6a98b83e0c6e68fe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:14.089694 7541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2da75716d0dc4df6a98b83e0c6e68fe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2da75716d0dc4df6a98b83e0c6e68fe7, State: Initialized, Role: FOLLOWER
I20260925 22:41:14.090181 7541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2da75716d0dc4df6a98b83e0c6e68fe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da75716d0dc4df6a98b83e0c6e68fe7" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 39123 } }
I20260925 22:41:14.090358 7541 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2da75716d0dc4df6a98b83e0c6e68fe7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:41:14.090436 7541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2da75716d0dc4df6a98b83e0c6e68fe7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:41:14.090565 7541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2da75716d0dc4df6a98b83e0c6e68fe7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:14.091743 7541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2da75716d0dc4df6a98b83e0c6e68fe7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da75716d0dc4df6a98b83e0c6e68fe7" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 39123 } }
I20260925 22:41:14.092083 2347 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:41:14.092168 7541 leader_election.cc:302] T 00000000000000000000000000000000 P 2da75716d0dc4df6a98b83e0c6e68fe7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2da75716d0dc4df6a98b83e0c6e68fe7; no voters:
I20260925 22:41:14.092553 7541 leader_election.cc:288] T 00000000000000000000000000000000 P 2da75716d0dc4df6a98b83e0c6e68fe7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:41:14.092685 7546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2da75716d0dc4df6a98b83e0c6e68fe7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:41:14.092957 7546 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2da75716d0dc4df6a98b83e0c6e68fe7 [term 1 LEADER]: Becoming Leader. State: Replica: 2da75716d0dc4df6a98b83e0c6e68fe7, State: Running, Role: LEADER
I20260925 22:41:14.093313 7546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2da75716d0dc4df6a98b83e0c6e68fe7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da75716d0dc4df6a98b83e0c6e68fe7" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 39123 } }
I20260925 22:41:14.093881 7541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2da75716d0dc4df6a98b83e0c6e68fe7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:41:14.094652 7548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2da75716d0dc4df6a98b83e0c6e68fe7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2da75716d0dc4df6a98b83e0c6e68fe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da75716d0dc4df6a98b83e0c6e68fe7" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 39123 } } }
I20260925 22:41:14.094974 7548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2da75716d0dc4df6a98b83e0c6e68fe7 [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:14.095108 7547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2da75716d0dc4df6a98b83e0c6e68fe7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2da75716d0dc4df6a98b83e0c6e68fe7. Latest consensus state: current_term: 1 leader_uuid: "2da75716d0dc4df6a98b83e0c6e68fe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da75716d0dc4df6a98b83e0c6e68fe7" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 39123 } } }
I20260925 22:41:14.095201 7547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2da75716d0dc4df6a98b83e0c6e68fe7 [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:14.095553 7555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:41:14.098726 7555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:41:14.104715 7555 catalog_manager.cc:1347] Generated new cluster ID: 6e9fc48a9802440da5f1e6800120cb0e
I20260925 22:41:14.104802 7555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:41:14.112603 7555 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:41:14.113867 7555 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:41:14.119148 7555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2da75716d0dc4df6a98b83e0c6e68fe7: Generated new TSK 0
I20260925 22:41:14.120043 7555 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 22:41:14.148953 7571 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 22:41:14.149091 7571 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 22:41:14.149847 7544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:14.132662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50732 (local address 127.2.74.254:39123)
0925 22:41:14.132800 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:14.132803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:14.133046 (+ 243us) server_negotiation.cc:407] Connection header received
0925 22:41:14.133318 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:14.133322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:14.133408 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:14.133526 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0925 22:41:14.133826 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:14.134392 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:14.135440 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:14.135681 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:14.135797 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:14.136211 (+ 414us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 22:41:14.149257 (+ 13046us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38695/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 22:41:14.149686 (+ 429us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50732: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38695/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 22:41:14.149928 7544 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50732: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38695/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 22:41:14.151185 7570 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:14.132555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:39123 (local address 127.0.0.1:50732)
0925 22:41:14.132939 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:14.132954 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:14.133132 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:14.133549 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:14.133552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:14.133563 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0925 22:41:14.133703 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:14.133709 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:14.134548 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:14.134552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:14.135292 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:14.135304 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:14.135813 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:14.135817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:14.135923 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:14.135979 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 22:41:14.149610 (+ 13631us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38695/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 22:41:14.149874 (+ 264us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.74.254:39123: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38695/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":132,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.74.254:39123: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38695/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 22:41:14.152179 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 7474
2026-09-25T22:41:14Z chronyd exiting
I20260925 22:41:14.189745 2347 test_util.cc:175] -----------------------------------------------
I20260925 22:41:14.189886 2347 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376047580784-2347-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (333 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 22:41:14.191111 2347 mini_oidc.cc:152] Starting JWKS server
I20260925 22:41:14.192022 2347 webserver.cc:460] Webserver started at https://127.0.0.1:41309/ using document root <none> and password file <none>
I20260925 22:41:14.192163 2347 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 22:41:14.192448 2347 webserver.cc:460] Webserver started at http://127.0.0.1:40683/ using document root <none> and password file <none>
2026-09-25T22:41:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:41:14Z Disabled control of system clock
I20260925 22:41:14.202111 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:35949
--webserver_interface=127.2.74.254
--webserver_port=0
--builtin_ntp_servers=127.2.74.212:40007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:35949
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41309/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376047580784-2347-0/testchainca.cert with env {}
W20260925 22:41:14.317884 7577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:14.318192 7577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:14.318236 7577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:14.322422 7577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:41:14.322506 7577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:14.322522 7577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:14.322540 7577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:41:14.322556 7577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 22:41:14.322572 7577 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 22:41:14.322587 7577 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41309/jwks.json
I20260925 22:41:14.328363 7577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:40007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:35949
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:35949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41309/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376047580784-2347-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:14.329500 7577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:14.330498 7577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:14.339615 7583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:14.339608 7582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:14.339618 7585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:14.339766 7577 server_base.cc:1034] running on GCE node
I20260925 22:41:14.340615 7577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:14.341841 7577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:14.343041 7577 hybrid_clock.cc:648] HybridClock initialized: now 1790376074343025 us; error 35 us; skew 500 ppm
I20260925 22:41:14.345091 7577 webserver.cc:460] Webserver started at http://127.2.74.254:37967/ using document root <none> and password file <none>
I20260925 22:41:14.345737 7577 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:14.345846 7577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:14.346092 7577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:14.348419 7577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "ed60c82ab4824a449992279ea7e6d08a"
format_stamp: "Formatted at 2026-09-25 22:41:14 on dist-test-slave-csnl"
server_key: "abdf9fd0988562ecb9f2986258893b24"
server_key_iv: "4eaf5a04bc361d94f56d57077ebf7b2a"
server_key_version: "encryptionkey@0"
I20260925 22:41:14.349078 7577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "ed60c82ab4824a449992279ea7e6d08a"
format_stamp: "Formatted at 2026-09-25 22:41:14 on dist-test-slave-csnl"
server_key: "abdf9fd0988562ecb9f2986258893b24"
server_key_iv: "4eaf5a04bc361d94f56d57077ebf7b2a"
server_key_version: "encryptionkey@0"
I20260925 22:41:14.353400 7577 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:41:14.356812 7591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:14.358279 7577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:41:14.358487 7577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "ed60c82ab4824a449992279ea7e6d08a"
format_stamp: "Formatted at 2026-09-25 22:41:14 on dist-test-slave-csnl"
server_key: "abdf9fd0988562ecb9f2986258893b24"
server_key_iv: "4eaf5a04bc361d94f56d57077ebf7b2a"
server_key_version: "encryptionkey@0"
I20260925 22:41:14.358640 7577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:14.374543 7577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:14.375430 7577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:14.375662 7577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:14.382851 7643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:35949 every 8 connection(s)
I20260925 22:41:14.382853 7577 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:35949
I20260925 22:41:14.384030 7577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:41:14.387578 7644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:14.388780 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 7577
I20260925 22:41:14.388904 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:41:14.389266 2347 external_mini_cluster.cc:1428] Setting key 81f5b5fab2af48c693d8b24872a3110e
I20260925 22:41:14.394532 7644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed60c82ab4824a449992279ea7e6d08a: Bootstrap starting.
I20260925 22:41:14.397074 7644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed60c82ab4824a449992279ea7e6d08a: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:14.397866 7644 log.cc:824] T 00000000000000000000000000000000 P ed60c82ab4824a449992279ea7e6d08a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:14.399326 7647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:14.390714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52808 (local address 127.2.74.254:35949)
0925 22:41:14.391489 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:14.391506 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:14.391547 (+ 41us) server_negotiation.cc:407] Connection header received
0925 22:41:14.392292 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:14.392325 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:14.392820 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:14.393199 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:14.393695 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:14.394731 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:14.395753 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:14.396263 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:14.396458 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:14.397402 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:14.397408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:14.397422 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:41:14.397832 (+ 410us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:14.397900 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:41:14.397920 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:14.398036 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:14.398162 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 22:41:14.398470 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":191,"threads_started":1}
I20260925 22:41:14.400189 7644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed60c82ab4824a449992279ea7e6d08a: No bootstrap required, opened a new log
I20260925 22:41:14.402384 2347 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:41:14.404500 7644 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed60c82ab4824a449992279ea7e6d08a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed60c82ab4824a449992279ea7e6d08a" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 35949 } }
I20260925 22:41:14.404804 7644 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed60c82ab4824a449992279ea7e6d08a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:14.404906 7644 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed60c82ab4824a449992279ea7e6d08a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed60c82ab4824a449992279ea7e6d08a, State: Initialized, Role: FOLLOWER
I20260925 22:41:14.405372 7644 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed60c82ab4824a449992279ea7e6d08a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed60c82ab4824a449992279ea7e6d08a" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 35949 } }
I20260925 22:41:14.405539 7644 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed60c82ab4824a449992279ea7e6d08a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:41:14.405625 7644 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed60c82ab4824a449992279ea7e6d08a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:41:14.405750 7644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed60c82ab4824a449992279ea7e6d08a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:14.406788 7644 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed60c82ab4824a449992279ea7e6d08a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed60c82ab4824a449992279ea7e6d08a" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 35949 } }
I20260925 22:41:14.407219 7644 leader_election.cc:302] T 00000000000000000000000000000000 P ed60c82ab4824a449992279ea7e6d08a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed60c82ab4824a449992279ea7e6d08a; no voters:
I20260925 22:41:14.407604 7644 leader_election.cc:288] T 00000000000000000000000000000000 P ed60c82ab4824a449992279ea7e6d08a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:41:14.407763 7649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed60c82ab4824a449992279ea7e6d08a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:41:14.408005 7649 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed60c82ab4824a449992279ea7e6d08a [term 1 LEADER]: Becoming Leader. State: Replica: ed60c82ab4824a449992279ea7e6d08a, State: Running, Role: LEADER
I20260925 22:41:14.408389 7649 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed60c82ab4824a449992279ea7e6d08a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed60c82ab4824a449992279ea7e6d08a" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 35949 } }
I20260925 22:41:14.408972 7644 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed60c82ab4824a449992279ea7e6d08a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:41:14.412173 7650 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed60c82ab4824a449992279ea7e6d08a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed60c82ab4824a449992279ea7e6d08a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed60c82ab4824a449992279ea7e6d08a" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 35949 } } }
I20260925 22:41:14.412209 7651 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed60c82ab4824a449992279ea7e6d08a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed60c82ab4824a449992279ea7e6d08a. Latest consensus state: current_term: 1 leader_uuid: "ed60c82ab4824a449992279ea7e6d08a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed60c82ab4824a449992279ea7e6d08a" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 35949 } } }
I20260925 22:41:14.412417 7650 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed60c82ab4824a449992279ea7e6d08a [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:14.412473 7651 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed60c82ab4824a449992279ea7e6d08a [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:14.413635 7663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:41:14.417300 7663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:41:14.423801 7663 catalog_manager.cc:1347] Generated new cluster ID: c0e49f99ea44486297aabc5754b8e55e
I20260925 22:41:14.423897 7663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:41:14.433837 7663 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:41:14.434949 7663 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:41:14.443526 7663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed60c82ab4824a449992279ea7e6d08a: Generated new TSK 0
I20260925 22:41:14.444407 7663 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 22:41:14.474278 7674 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 22:41:14.474380 7674 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 22:41:14.475219 7647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:14.458517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52820 (local address 127.2.74.254:35949)
0925 22:41:14.458675 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:14.458681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:14.458869 (+ 188us) server_negotiation.cc:407] Connection header received
0925 22:41:14.459079 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:14.459082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:14.459136 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:14.459236 (+ 100us) server_negotiation.cc:226] Negotiated authn=JWT
0925 22:41:14.459561 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:14.460111 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:14.461483 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:14.461723 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:14.461864 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:14.462288 (+ 424us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 22:41:14.474722 (+ 12434us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41309/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 22:41:14.475061 (+ 339us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52820: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41309/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 22:41:14.475312 7673 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:14.458422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:35949 (local address 127.0.0.1:52820)
0925 22:41:14.458766 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:14.458783 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:14.458911 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:14.459267 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:14.459271 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:14.459279 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0925 22:41:14.459440 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:14.459447 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:14.460237 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:14.460241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:14.461311 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:14.461325 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:14.461894 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:14.461898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:14.462014 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:14.462047 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 22:41:14.474944 (+ 12897us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41309/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 22:41:14.475146 (+ 202us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.74.254:35949: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41309/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":120,"threads_started":1}
W20260925 22:41:14.475310 7647 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52820: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41309/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260925 22:41:14.476120 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 7577
2026-09-25T22:41:14Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (324 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 22:41:14.515713 2347 mini_oidc.cc:152] Starting JWKS server
I20260925 22:41:14.516626 2347 webserver.cc:460] Webserver started at https://127.0.0.1:46427/ using document root <none> and password file <none>
I20260925 22:41:14.516767 2347 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 22:41:14.517061 2347 webserver.cc:460] Webserver started at http://127.0.0.1:40443/ using document root <none> and password file <none>
2026-09-25T22:41:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:41:14Z Disabled control of system clock
I20260925 22:41:14.526607 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:33029
--webserver_interface=127.2.74.254
--webserver_port=0
--builtin_ntp_servers=127.2.74.212:40153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:33029
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46427/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 22:41:14.639652 7680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:14.639953 7680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:14.639993 7680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:14.644200 7680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:41:14.644270 7680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:14.644285 7680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:14.644304 7680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:41:14.644320 7680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 22:41:14.644389 7680 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 22:41:14.644419 7680 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46427/jwks.json
I20260925 22:41:14.649441 7680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:40153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:33029
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:33029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46427/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:14.650714 7680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:14.652173 7680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:14.660861 7688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:14.660890 7686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:14.661012 7680 server_base.cc:1034] running on GCE node
W20260925 22:41:14.660861 7685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:14.661870 7680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:14.663105 7680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:14.664294 7680 hybrid_clock.cc:648] HybridClock initialized: now 1790376074664273 us; error 36 us; skew 500 ppm
I20260925 22:41:14.666277 7680 webserver.cc:460] Webserver started at http://127.2.74.254:39551/ using document root <none> and password file <none>
I20260925 22:41:14.666895 7680 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:14.667030 7680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:14.667274 7680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:14.669404 7680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "1c8d52ee75d7416cbcdcc4a08df0f9e2"
format_stamp: "Formatted at 2026-09-25 22:41:14 on dist-test-slave-csnl"
server_key: "e5ae8557bd47e6e18daa59c500aeeb41"
server_key_iv: "a3fe7264622cfa7937ba0f3786fc3de6"
server_key_version: "encryptionkey@0"
I20260925 22:41:14.670014 7680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "1c8d52ee75d7416cbcdcc4a08df0f9e2"
format_stamp: "Formatted at 2026-09-25 22:41:14 on dist-test-slave-csnl"
server_key: "e5ae8557bd47e6e18daa59c500aeeb41"
server_key_iv: "a3fe7264622cfa7937ba0f3786fc3de6"
server_key_version: "encryptionkey@0"
I20260925 22:41:14.673935 7680 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 22:41:14.676705 7694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:14.677999 7680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:41:14.678153 7680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "1c8d52ee75d7416cbcdcc4a08df0f9e2"
format_stamp: "Formatted at 2026-09-25 22:41:14 on dist-test-slave-csnl"
server_key: "e5ae8557bd47e6e18daa59c500aeeb41"
server_key_iv: "a3fe7264622cfa7937ba0f3786fc3de6"
server_key_version: "encryptionkey@0"
I20260925 22:41:14.678282 7680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:14.691643 7680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:14.692407 7680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:14.692631 7680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:14.700101 7680 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:33029
I20260925 22:41:14.700103 7746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:33029 every 8 connection(s)
I20260925 22:41:14.701174 7680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:41:14.702744 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 7680
I20260925 22:41:14.702989 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:41:14.703361 2347 external_mini_cluster.cc:1428] Setting key cf84af7d976dcccba78073ef2a84c16b
I20260925 22:41:14.705120 7747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:14.711248 7747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c8d52ee75d7416cbcdcc4a08df0f9e2: Bootstrap starting.
I20260925 22:41:14.713258 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:14.704886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60422 (local address 127.2.74.254:33029)
0925 22:41:14.705629 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:14.705647 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:14.705684 (+ 37us) server_negotiation.cc:407] Connection header received
0925 22:41:14.706382 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:14.706400 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:14.706837 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:14.707291 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:14.707755 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:14.708775 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:14.709707 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:14.710206 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:14.710400 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:14.711398 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:14.711404 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:14.711419 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:41:14.711833 (+ 414us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:14.711891 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:41:14.711911 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:14.712031 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:14.712192 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 22:41:14.712477 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":144,"threads_started":1}
I20260925 22:41:14.713830 7747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c8d52ee75d7416cbcdcc4a08df0f9e2: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:14.714622 7747 log.cc:824] T 00000000000000000000000000000000 P 1c8d52ee75d7416cbcdcc4a08df0f9e2: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:14.716480 2347 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:41:14.717034 7747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c8d52ee75d7416cbcdcc4a08df0f9e2: No bootstrap required, opened a new log
I20260925 22:41:14.721100 7747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c8d52ee75d7416cbcdcc4a08df0f9e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8d52ee75d7416cbcdcc4a08df0f9e2" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33029 } }
I20260925 22:41:14.721410 7747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c8d52ee75d7416cbcdcc4a08df0f9e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:14.721450 7747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c8d52ee75d7416cbcdcc4a08df0f9e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c8d52ee75d7416cbcdcc4a08df0f9e2, State: Initialized, Role: FOLLOWER
I20260925 22:41:14.721946 7747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c8d52ee75d7416cbcdcc4a08df0f9e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8d52ee75d7416cbcdcc4a08df0f9e2" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33029 } }
I20260925 22:41:14.722072 7747 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c8d52ee75d7416cbcdcc4a08df0f9e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:41:14.722163 7747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c8d52ee75d7416cbcdcc4a08df0f9e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:41:14.722283 7747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c8d52ee75d7416cbcdcc4a08df0f9e2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:14.723433 7747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c8d52ee75d7416cbcdcc4a08df0f9e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8d52ee75d7416cbcdcc4a08df0f9e2" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33029 } }
I20260925 22:41:14.723815 7747 leader_election.cc:302] T 00000000000000000000000000000000 P 1c8d52ee75d7416cbcdcc4a08df0f9e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c8d52ee75d7416cbcdcc4a08df0f9e2; no voters:
I20260925 22:41:14.724180 7747 leader_election.cc:288] T 00000000000000000000000000000000 P 1c8d52ee75d7416cbcdcc4a08df0f9e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:41:14.724318 7752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c8d52ee75d7416cbcdcc4a08df0f9e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:41:14.724596 7752 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c8d52ee75d7416cbcdcc4a08df0f9e2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c8d52ee75d7416cbcdcc4a08df0f9e2, State: Running, Role: LEADER
I20260925 22:41:14.724979 7752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c8d52ee75d7416cbcdcc4a08df0f9e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8d52ee75d7416cbcdcc4a08df0f9e2" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33029 } }
I20260925 22:41:14.725534 7747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c8d52ee75d7416cbcdcc4a08df0f9e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:41:14.726691 7754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c8d52ee75d7416cbcdcc4a08df0f9e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c8d52ee75d7416cbcdcc4a08df0f9e2. Latest consensus state: current_term: 1 leader_uuid: "1c8d52ee75d7416cbcdcc4a08df0f9e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8d52ee75d7416cbcdcc4a08df0f9e2" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33029 } } }
I20260925 22:41:14.726845 7754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c8d52ee75d7416cbcdcc4a08df0f9e2 [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:14.727166 7753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c8d52ee75d7416cbcdcc4a08df0f9e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c8d52ee75d7416cbcdcc4a08df0f9e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8d52ee75d7416cbcdcc4a08df0f9e2" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33029 } } }
I20260925 22:41:14.727252 7761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:41:14.727264 7753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c8d52ee75d7416cbcdcc4a08df0f9e2 [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:14.730777 7761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:41:14.737426 7761 catalog_manager.cc:1347] Generated new cluster ID: b6655751e3d745309c7f3fad039980af
I20260925 22:41:14.737541 7761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:41:14.748294 7761 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:41:14.749281 7761 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:41:14.756448 7761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c8d52ee75d7416cbcdcc4a08df0f9e2: Generated new TSK 0
I20260925 22:41:14.757232 7761 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:41:14.775404 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:14.770302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60424 (local address 127.2.74.254:33029)
0925 22:41:14.770452 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:14.770457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:14.770633 (+ 176us) server_negotiation.cc:407] Connection header received
0925 22:41:14.770805 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:14.770808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:14.770858 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:14.770975 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0925 22:41:14.771257 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:14.771793 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:14.773035 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:14.773252 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:14.773335 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:14.773671 (+ 336us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 22:41:14.775006 (+ 1335us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46427/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 22:41:14.775271 (+ 265us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60424: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46427/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 22:41:14.775496 7750 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60424: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46427/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 22:41:14.778218 7776 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:14.770207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33029 (local address 127.0.0.1:60424)
0925 22:41:14.770534 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:14.770549 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:14.770674 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:14.771001 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:14.771004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:14.771013 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 22:41:14.771147 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:14.771153 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:14.771909 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:14.771912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:14.772907 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:14.772918 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:14.773382 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:14.773386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:14.773480 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:14.773510 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 22:41:14.775176 (+ 1666us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46427/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 22:41:14.778072 (+ 2896us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.74.254:33029: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46427/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":2690,"thread_start_us":117,"threads_started":1}
I20260925 22:41:14.778857 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 7680
E20260925 22:41:14.789800 7777 webserver.cc:533] Webserver: sslize failed:
E20260925 22:41:14.789891 7777 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T22:41:14Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (303 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T22:41:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:41:14Z Disabled control of system clock
I20260925 22:41:14.828068 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:46069
--webserver_interface=127.2.74.254
--webserver_port=0
--builtin_ntp_servers=127.2.74.212:37711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:46069
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376047580784-2347-0/testchainca.cert with env {}
W20260925 22:41:14.946755 7781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:14.947121 7781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:14.947170 7781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:14.951434 7781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:41:14.951511 7781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:14.951527 7781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:14.951545 7781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:41:14.951598 7781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 22:41:14.951628 7781 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 22:41:14.951660 7781 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 22:41:14.957587 7781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:37711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:46069
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:46069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376047580784-2347-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:14.958859 7781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:14.959975 7781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:14.969341 7787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:14.969344 7786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:14.969400 7789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:14.969574 7781 server_base.cc:1034] running on GCE node
I20260925 22:41:14.970458 7781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:14.971796 7781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:14.973001 7781 hybrid_clock.cc:648] HybridClock initialized: now 1790376074972981 us; error 41 us; skew 500 ppm
I20260925 22:41:14.975157 7781 webserver.cc:460] Webserver started at http://127.2.74.254:39009/ using document root <none> and password file <none>
I20260925 22:41:14.975785 7781 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:14.975853 7781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:14.976111 7781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:14.978361 7781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "100c74f492b34eb795a11ede58a2c4da"
format_stamp: "Formatted at 2026-09-25 22:41:14 on dist-test-slave-csnl"
server_key: "3a2789eb993ac8c1454885ae737e4e6f"
server_key_iv: "e6654a765977ce1048460d7a2e8490ab"
server_key_version: "encryptionkey@0"
I20260925 22:41:14.979077 7781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "100c74f492b34eb795a11ede58a2c4da"
format_stamp: "Formatted at 2026-09-25 22:41:14 on dist-test-slave-csnl"
server_key: "3a2789eb993ac8c1454885ae737e4e6f"
server_key_iv: "e6654a765977ce1048460d7a2e8490ab"
server_key_version: "encryptionkey@0"
I20260925 22:41:14.983201 7781 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:41:14.985987 7795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:14.987404 7781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:41:14.987560 7781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "100c74f492b34eb795a11ede58a2c4da"
format_stamp: "Formatted at 2026-09-25 22:41:14 on dist-test-slave-csnl"
server_key: "3a2789eb993ac8c1454885ae737e4e6f"
server_key_iv: "e6654a765977ce1048460d7a2e8490ab"
server_key_version: "encryptionkey@0"
I20260925 22:41:14.987691 7781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:15.002202 7781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:15.002977 7781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:15.003185 7781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:15.010519 7781 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:46069
I20260925 22:41:15.010545 7847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:46069 every 8 connection(s)
I20260925 22:41:15.011602 7781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:41:15.015017 7848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:15.015373 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 7781
I20260925 22:41:15.015528 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:41:15.015846 2347 external_mini_cluster.cc:1428] Setting key 100da3c1b310e2eb6f62af8459546445
I20260925 22:41:15.022862 7848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 100c74f492b34eb795a11ede58a2c4da: Bootstrap starting.
I20260925 22:41:15.025637 7848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 100c74f492b34eb795a11ede58a2c4da: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:15.025961 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:15.017402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56676 (local address 127.2.74.254:46069)
0925 22:41:15.018157 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:15.018173 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:15.018212 (+ 39us) server_negotiation.cc:407] Connection header received
0925 22:41:15.018919 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:15.018942 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:15.019476 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:15.019785 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:15.020213 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:15.021208 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:15.022472 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:15.023006 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:15.023156 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:15.024073 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:15.024078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:15.024091 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:41:15.024502 (+ 411us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:15.024560 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:41:15.024581 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:15.024660 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:15.024830 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 22:41:15.025147 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":159,"threads_started":1}
I20260925 22:41:15.026587 7848 log.cc:824] T 00000000000000000000000000000000 P 100c74f492b34eb795a11ede58a2c4da: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:15.029019 7848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 100c74f492b34eb795a11ede58a2c4da: No bootstrap required, opened a new log
I20260925 22:41:15.029307 2347 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:41:15.033294 7848 raft_consensus.cc:348] T 00000000000000000000000000000000 P 100c74f492b34eb795a11ede58a2c4da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100c74f492b34eb795a11ede58a2c4da" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 46069 } }
I20260925 22:41:15.033593 7848 raft_consensus.cc:374] T 00000000000000000000000000000000 P 100c74f492b34eb795a11ede58a2c4da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:15.033661 7848 raft_consensus.cc:729] T 00000000000000000000000000000000 P 100c74f492b34eb795a11ede58a2c4da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100c74f492b34eb795a11ede58a2c4da, State: Initialized, Role: FOLLOWER
I20260925 22:41:15.034125 7848 consensus_queue.cc:260] T 00000000000000000000000000000000 P 100c74f492b34eb795a11ede58a2c4da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100c74f492b34eb795a11ede58a2c4da" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 46069 } }
I20260925 22:41:15.034289 7848 raft_consensus.cc:388] T 00000000000000000000000000000000 P 100c74f492b34eb795a11ede58a2c4da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:41:15.034363 7848 raft_consensus.cc:482] T 00000000000000000000000000000000 P 100c74f492b34eb795a11ede58a2c4da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:41:15.034508 7848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 100c74f492b34eb795a11ede58a2c4da [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:15.035647 7848 raft_consensus.cc:504] T 00000000000000000000000000000000 P 100c74f492b34eb795a11ede58a2c4da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100c74f492b34eb795a11ede58a2c4da" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 46069 } }
I20260925 22:41:15.036088 7848 leader_election.cc:302] T 00000000000000000000000000000000 P 100c74f492b34eb795a11ede58a2c4da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 100c74f492b34eb795a11ede58a2c4da; no voters:
I20260925 22:41:15.036521 7848 leader_election.cc:288] T 00000000000000000000000000000000 P 100c74f492b34eb795a11ede58a2c4da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:41:15.036657 7853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 100c74f492b34eb795a11ede58a2c4da [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:41:15.036896 7853 raft_consensus.cc:686] T 00000000000000000000000000000000 P 100c74f492b34eb795a11ede58a2c4da [term 1 LEADER]: Becoming Leader. State: Replica: 100c74f492b34eb795a11ede58a2c4da, State: Running, Role: LEADER
I20260925 22:41:15.037228 7853 consensus_queue.cc:237] T 00000000000000000000000000000000 P 100c74f492b34eb795a11ede58a2c4da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100c74f492b34eb795a11ede58a2c4da" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 46069 } }
I20260925 22:41:15.037887 7848 sys_catalog.cc:564] T 00000000000000000000000000000000 P 100c74f492b34eb795a11ede58a2c4da [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:41:15.039034 7855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 100c74f492b34eb795a11ede58a2c4da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 100c74f492b34eb795a11ede58a2c4da. Latest consensus state: current_term: 1 leader_uuid: "100c74f492b34eb795a11ede58a2c4da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100c74f492b34eb795a11ede58a2c4da" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 46069 } } }
I20260925 22:41:15.039193 7855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 100c74f492b34eb795a11ede58a2c4da [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:15.039474 7854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 100c74f492b34eb795a11ede58a2c4da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "100c74f492b34eb795a11ede58a2c4da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100c74f492b34eb795a11ede58a2c4da" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 46069 } } }
I20260925 22:41:15.039569 7854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 100c74f492b34eb795a11ede58a2c4da [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:15.040304 7862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:41:15.043248 7862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:41:15.048897 7862 catalog_manager.cc:1347] Generated new cluster ID: b5580d5c8bc44aed871bfc4b4cadc712
I20260925 22:41:15.048998 7862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:41:15.058727 7862 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:41:15.059768 7862 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:41:15.072956 7862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 100c74f492b34eb795a11ede58a2c4da: Generated new TSK 0
I20260925 22:41:15.073756 7862 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:41:15.169701 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:15.085504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56680 (local address 127.2.74.254:46069)
0925 22:41:15.085659 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:15.085664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:15.085839 (+ 175us) server_negotiation.cc:407] Connection header received
0925 22:41:15.086036 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:15.086038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:15.086092 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:15.086182 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0925 22:41:15.086471 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:15.087080 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:15.088382 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:15.088619 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:15.088700 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:15.089080 (+ 380us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 22:41:15.169300 (+ 80220us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 22:41:15.169579 (+ 279us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56680: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 22:41:15.169813 7851 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56680: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 22:41:15.170271 7877 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:15.085392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:46069 (local address 127.0.0.1:56680)
0925 22:41:15.085735 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:15.085751 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:15.085882 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:15.086207 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:15.086210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:15.086219 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 22:41:15.086344 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:15.086351 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:15.087203 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:15.087208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:15.088236 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:15.088249 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:15.088752 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:15.088756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:15.088858 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:15.088891 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 22:41:15.169556 (+ 80665us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 22:41:15.169768 (+ 212us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.74.254:46069: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":139,"threads_started":1}
I20260925 22:41:15.170915 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 7781
2026-09-25T22:41:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (371 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (127 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (131 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:41:15 dist-test-slave-csnl krb5kdc[7889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:41:15 dist-test-slave-csnl krb5kdc[7889](info): set up 2 sockets
Sep 25 22:41:15 dist-test-slave-csnl krb5kdc[7889](info): commencing operation
krb5kdc: starting...
W20260925 22:41:16.467401 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:41:16 dist-test-slave-csnl krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:41:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:41:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:41:16.593436 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:40503
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:45565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:40503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:16.709846 7902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:16.710156 7902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:16.710199 7902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:16.714497 7902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:41:16.714587 7902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:16.714607 7902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:16.714624 7902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:41:16.714640 7902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:41:16.720518 7902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:45565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:40503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:40503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:16.721673 7902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:16.722718 7902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:16.731856 7910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:16.731840 7908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:16.731840 7907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:16.732918 7902 server_base.cc:1034] running on GCE node
I20260925 22:41:16.733558 7902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:16.734783 7902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:16.735978 7902 hybrid_clock.cc:648] HybridClock initialized: now 1790376076735969 us; error 34 us; skew 500 ppm
Sep 25 22:41:16 dist-test-slave-csnl krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376076, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:16.738922 7902 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:41:16.739959 7902 webserver.cc:460] Webserver started at http://127.2.74.254:32821/ using document root <none> and password file <none>
I20260925 22:41:16.740566 7902 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:16.740664 7902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:16.740904 7902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:16.743129 7902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "10fcd47220b04629863d426cf8c23402"
format_stamp: "Formatted at 2026-09-25 22:41:16 on dist-test-slave-csnl"
server_key: "2d38e55dc3b2d9d9fec8793143d88d84"
server_key_iv: "49e765e36e641a8e7bf001116e25bf6d"
server_key_version: "encryptionkey@0"
I20260925 22:41:16.743726 7902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "10fcd47220b04629863d426cf8c23402"
format_stamp: "Formatted at 2026-09-25 22:41:16 on dist-test-slave-csnl"
server_key: "2d38e55dc3b2d9d9fec8793143d88d84"
server_key_iv: "49e765e36e641a8e7bf001116e25bf6d"
server_key_version: "encryptionkey@0"
I20260925 22:41:16.747790 7902 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:41:16.750416 7917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:16.751788 7902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:41:16.751940 7902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "10fcd47220b04629863d426cf8c23402"
format_stamp: "Formatted at 2026-09-25 22:41:16 on dist-test-slave-csnl"
server_key: "2d38e55dc3b2d9d9fec8793143d88d84"
server_key_iv: "49e765e36e641a8e7bf001116e25bf6d"
server_key_version: "encryptionkey@0"
I20260925 22:41:16.752066 7902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:16.770051 7902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:16.773162 7902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:16.773396 7902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:16.780750 7902 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:40503
I20260925 22:41:16.780768 7969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:40503 every 8 connection(s)
I20260925 22:41:16.781808 7902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:41:16.784711 7970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:16.790529 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 7902
I20260925 22:41:16.790658 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:41:16.790776 7970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10fcd47220b04629863d426cf8c23402: Bootstrap starting.
I20260925 22:41:16.791002 2347 external_mini_cluster.cc:1428] Setting key 0712cf77e998f3f3d4e2531b69f2a7ae
I20260925 22:41:16.793438 7970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10fcd47220b04629863d426cf8c23402: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:16.794266 7970 log.cc:824] T 00000000000000000000000000000000 P 10fcd47220b04629863d426cf8c23402: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:16.796368 7970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10fcd47220b04629863d426cf8c23402: No bootstrap required, opened a new log
Sep 25 22:41:16 dist-test-slave-csnl krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:41:16.800796 7970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10fcd47220b04629863d426cf8c23402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10fcd47220b04629863d426cf8c23402" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40503 } }
I20260925 22:41:16.801069 7970 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10fcd47220b04629863d426cf8c23402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:16.801159 7970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10fcd47220b04629863d426cf8c23402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10fcd47220b04629863d426cf8c23402, State: Initialized, Role: FOLLOWER
I20260925 22:41:16.801705 7970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10fcd47220b04629863d426cf8c23402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10fcd47220b04629863d426cf8c23402" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40503 } }
I20260925 22:41:16.801893 7970 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10fcd47220b04629863d426cf8c23402 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:41:16.802004 7970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10fcd47220b04629863d426cf8c23402 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:41:16.802114 7970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10fcd47220b04629863d426cf8c23402 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:16.803102 7970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10fcd47220b04629863d426cf8c23402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10fcd47220b04629863d426cf8c23402" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40503 } }
I20260925 22:41:16.803506 7970 leader_election.cc:302] T 00000000000000000000000000000000 P 10fcd47220b04629863d426cf8c23402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10fcd47220b04629863d426cf8c23402; no voters:
I20260925 22:41:16.803905 7970 leader_election.cc:288] T 00000000000000000000000000000000 P 10fcd47220b04629863d426cf8c23402 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:41:16.804024 7975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10fcd47220b04629863d426cf8c23402 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:41:16.804278 7975 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10fcd47220b04629863d426cf8c23402 [term 1 LEADER]: Becoming Leader. State: Replica: 10fcd47220b04629863d426cf8c23402, State: Running, Role: LEADER
I20260925 22:41:16.804674 7975 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10fcd47220b04629863d426cf8c23402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10fcd47220b04629863d426cf8c23402" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40503 } }
I20260925 22:41:16.805455 7970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10fcd47220b04629863d426cf8c23402 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:41:16.806591 7976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10fcd47220b04629863d426cf8c23402 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10fcd47220b04629863d426cf8c23402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10fcd47220b04629863d426cf8c23402" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40503 } } }
I20260925 22:41:16.806586 7977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10fcd47220b04629863d426cf8c23402 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10fcd47220b04629863d426cf8c23402. Latest consensus state: current_term: 1 leader_uuid: "10fcd47220b04629863d426cf8c23402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10fcd47220b04629863d426cf8c23402" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40503 } } }
I20260925 22:41:16.806736 7977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10fcd47220b04629863d426cf8c23402 [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:16.806736 7976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10fcd47220b04629863d426cf8c23402 [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:16.806823 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:16.792329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43774 (local address 127.2.74.254:40503)
0925 22:41:16.792750 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:16.792769 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:16.792816 (+ 47us) server_negotiation.cc:407] Connection header received
0925 22:41:16.793460 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:16.793477 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:16.793890 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:16.794174 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:16.795134 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:16.796105 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:16.797257 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:16.797745 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:16.797886 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:16.800797 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:16.800803 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:16.800817 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:16.800846 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:16.802755 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:16.803349 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:16.803355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:16.803362 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:16.803521 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:16.803898 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:16.803901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:16.803903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:16.804478 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:16.805256 (+ 778us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:16.805411 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 22:41:16.805745 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":145,"threads_started":1}
I20260925 22:41:16.807178 7984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:41:16.810267 7984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:41:16.816171 7984 catalog_manager.cc:1347] Generated new cluster ID: 0b5806c177ec4a83b7392273e2eda128
I20260925 22:41:16.816264 7984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:41:16.824792 7984 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:41:16.826215 7984 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:41:16.838423 7984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10fcd47220b04629863d426cf8c23402: Generated new TSK 0
I20260925 22:41:16.839504 7984 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
I20260925 22:41:16.868404 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:0
--local_ip_for_outbound_sockets=127.2.74.193
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:40503
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:45565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:16.987602 7996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:16.987938 7996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:16.988013 7996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:16.992187 7996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:16.992300 7996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:16.992396 7996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:41:16.997957 7996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:45565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:40503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:16.999131 7996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:17.000200 7996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:17.010774 8002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:17.010820 8001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:17.010769 8004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:17.011210 7996 server_base.cc:1034] running on GCE node
I20260925 22:41:17.011698 7996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:17.012475 7996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:17.013659 7996 hybrid_clock.cc:648] HybridClock initialized: now 1790376077013641 us; error 35 us; skew 500 ppm
Sep 25 22:41:17 dist-test-slave-csnl krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376077, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:17.016974 7996 init.cc:377] Logged in from keytab as kudu/127.2.74.193@KRBTEST.COM (short username kudu)
I20260925 22:41:17.018054 7996 webserver.cc:460] Webserver started at http://127.2.74.193:42359/ using document root <none> and password file <none>
I20260925 22:41:17.018712 7996 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:17.018786 7996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:17.019116 7996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:17.021420 7996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-0/data/instance:
uuid: "ff1a16b9b4e148f8a3ec02c66c69a268"
format_stamp: "Formatted at 2026-09-25 22:41:17 on dist-test-slave-csnl"
server_key: "dc9a01777b1a81386cca15bdd0da187a"
server_key_iv: "5e11f9abd4e9637a1f7a59baea339a62"
server_key_version: "encryptionkey@0"
I20260925 22:41:17.022085 7996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance:
uuid: "ff1a16b9b4e148f8a3ec02c66c69a268"
format_stamp: "Formatted at 2026-09-25 22:41:17 on dist-test-slave-csnl"
server_key: "dc9a01777b1a81386cca15bdd0da187a"
server_key_iv: "5e11f9abd4e9637a1f7a59baea339a62"
server_key_version: "encryptionkey@0"
I20260925 22:41:17.026472 7996 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260925 22:41:17.029349 8011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:17.030844 7996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:41:17.031054 7996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "ff1a16b9b4e148f8a3ec02c66c69a268"
format_stamp: "Formatted at 2026-09-25 22:41:17 on dist-test-slave-csnl"
server_key: "dc9a01777b1a81386cca15bdd0da187a"
server_key_iv: "5e11f9abd4e9637a1f7a59baea339a62"
server_key_version: "encryptionkey@0"
I20260925 22:41:17.031201 7996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:17.046132 7996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:17.049856 7996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:17.050086 7996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:17.050740 7996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:17.051766 7996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:17.051841 7996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:17.051908 7996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:17.051947 7996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:17.061964 7996 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:45007
I20260925 22:41:17.061983 8124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:45007 every 8 connection(s)
I20260925 22:41:17.063038 7996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:41:17.066182 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 7996
I20260925 22:41:17.066321 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance
I20260925 22:41:17.066627 2347 external_mini_cluster.cc:1428] Setting key f6b02b5d5130ab1246e03f97faf03250
Sep 25 22:41:17 dist-test-slave-csnl krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376077, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.74.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.194@KRBTEST.COM" created.
I20260925 22:41:17.080070 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:17.065184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:52169 (local address 127.2.74.254:40503)
0925 22:41:17.065381 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:17.065386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:17.066335 (+ 949us) server_negotiation.cc:407] Connection header received
0925 22:41:17.068104 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:17.068109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:17.068160 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:17.068243 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:17.069711 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:17.070230 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:17.072379 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:17.072691 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:17.072782 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:17.076170 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:17.076173 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:17.076175 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:17.076201 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:17.077729 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:17.078327 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:17.078330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:17.078331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:17.078418 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:17.078793 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:17.078796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:17.078797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:17.079065 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:17.079178 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:17.079759 (+ 581us) server_negotiation.cc:299] Negotiation successful
0925 22:41:17.079896 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 22:41:17.080963 8127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:17.065595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:40503 (local address 127.2.74.193:52169)
0925 22:41:17.066113 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:17.066165 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:17.067826 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:17.068461 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:17.068469 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:17.068885 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:17.069501 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:17.069527 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:17.070380 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:17.070386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:17.072220 (+ 1834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:17.072237 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:17.072894 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:17.072900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:17.073161 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:17.073921 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:17.073954 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:17.075948 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:17.077872 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:17.077879 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:17.077899 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:17.078200 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:17.078526 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:17.078530 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:17.078532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:17.078690 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:17.079178 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:17.079186 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:17.079466 (+ 280us) client_negotiation.cc:769] Sending connection context
0925 22:41:17.079706 (+ 240us) client_negotiation.cc:240] Negotiation successful
0925 22:41:17.080075 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":134,"threads_started":1}
I20260925 22:41:17.082628 8125 heartbeater.cc:347] Connected to a master server at 127.2.74.254:40503
I20260925 22:41:17.082911 8125 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:17.083545 8125 heartbeater.cc:511] Master 127.2.74.254:40503 requested a full tablet report, sending...
I20260925 22:41:17.085158 7934 ts_manager.cc:194] Registered new tserver with Master: ff1a16b9b4e148f8a3ec02c66c69a268 (127.2.74.193:45007)
I20260925 22:41:17.086339 7934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.193@KRBTEST.COM'} at 127.2.74.193:52169
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
I20260925 22:41:17.098222 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:0
--local_ip_for_outbound_sockets=127.2.74.194
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:40503
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:45565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:17.215183 8130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:17.215467 8130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:17.215513 8130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:17.219789 8130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:17.219868 8130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:17.219931 8130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:41:17.225584 8130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:45565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:40503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:17.226688 8130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:17.227777 8130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:17.238557 8136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:17.238557 8135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:17.238641 8130 server_base.cc:1034] running on GCE node
W20260925 22:41:17.238559 8138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:17.239396 8130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:17.240100 8130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:17.241330 8130 hybrid_clock.cc:648] HybridClock initialized: now 1790376077241301 us; error 39 us; skew 500 ppm
Sep 25 22:41:17 dist-test-slave-csnl krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376077, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:17.244537 8130 init.cc:377] Logged in from keytab as kudu/127.2.74.194@KRBTEST.COM (short username kudu)
I20260925 22:41:17.245507 8130 webserver.cc:460] Webserver started at http://127.2.74.194:46451/ using document root <none> and password file <none>
I20260925 22:41:17.246064 8130 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:17.246129 8130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:17.246331 8130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:17.248575 8130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-1/data/instance:
uuid: "85dc5621eebb4213b8ee68d8a12698c8"
format_stamp: "Formatted at 2026-09-25 22:41:17 on dist-test-slave-csnl"
server_key: "81a19044d932b16188343caa3b17d0a8"
server_key_iv: "99729338df1d5a3fe2c48b3109037fd9"
server_key_version: "encryptionkey@0"
I20260925 22:41:17.249142 8130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance:
uuid: "85dc5621eebb4213b8ee68d8a12698c8"
format_stamp: "Formatted at 2026-09-25 22:41:17 on dist-test-slave-csnl"
server_key: "81a19044d932b16188343caa3b17d0a8"
server_key_iv: "99729338df1d5a3fe2c48b3109037fd9"
server_key_version: "encryptionkey@0"
I20260925 22:41:17.253283 8130 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:41:17.256214 8145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:17.257650 8130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:41:17.257767 8130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "85dc5621eebb4213b8ee68d8a12698c8"
format_stamp: "Formatted at 2026-09-25 22:41:17 on dist-test-slave-csnl"
server_key: "81a19044d932b16188343caa3b17d0a8"
server_key_iv: "99729338df1d5a3fe2c48b3109037fd9"
server_key_version: "encryptionkey@0"
I20260925 22:41:17.257876 8130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:17.278489 8130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:17.281832 8130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:17.282073 8130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:17.282725 8130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:17.283727 8130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:17.283811 8130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:17.283891 8130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:17.283922 8130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:17.293488 8130 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:38447
I20260925 22:41:17.293532 8258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:38447 every 8 connection(s)
I20260925 22:41:17.294493 8130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:41:17.295468 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 8130
I20260925 22:41:17.295609 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance
I20260925 22:41:17.295895 2347 external_mini_cluster.cc:1428] Setting key ab8bba6ef3189b4ba21e1680113dfa82
WARNING: no policy specified for kudu/127.2.74.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.195@KRBTEST.COM" created.
Sep 25 22:41:17 dist-test-slave-csnl krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376077, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:41:17.313030 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:17.296619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:36341 (local address 127.2.74.254:40503)
0925 22:41:17.296774 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:17.296779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:17.297831 (+ 1052us) server_negotiation.cc:407] Connection header received
0925 22:41:17.299904 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:17.299907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:17.299990 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:17.300097 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:17.301443 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:17.301984 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:17.303291 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:17.303524 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:17.303640 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:17.308846 (+ 5206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:17.308854 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:17.308860 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:17.308902 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:17.310489 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:17.311304 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:17.311312 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:17.311314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:17.311404 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:17.311806 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:17.311810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:17.311812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:17.312023 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:17.312138 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:17.312700 (+ 562us) server_negotiation.cc:299] Negotiation successful
0925 22:41:17.312838 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 22:41:17.313922 8261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:17.297135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:40503 (local address 127.2.74.194:36341)
0925 22:41:17.297651 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:17.297711 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:17.299620 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:17.300276 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:17.300284 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:17.300707 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:17.301230 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:17.301244 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:17.302129 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:17.302138 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:17.303147 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:17.303157 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:17.303704 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:17.303708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:17.303889 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:17.304567 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:17.304584 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:17.308610 (+ 4026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:17.310683 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:17.310691 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:17.310713 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:17.311169 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:17.311522 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:17.311526 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:17.311528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:17.311696 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:17.312137 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:17.312146 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:17.312433 (+ 287us) client_negotiation.cc:769] Sending connection context
0925 22:41:17.312645 (+ 212us) client_negotiation.cc:240] Negotiation successful
0925 22:41:17.313020 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":154,"threads_started":1}
I20260925 22:41:17.315629 8259 heartbeater.cc:347] Connected to a master server at 127.2.74.254:40503
I20260925 22:41:17.315891 8259 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:17.316424 8259 heartbeater.cc:511] Master 127.2.74.254:40503 requested a full tablet report, sending...
I20260925 22:41:17.317551 7934 ts_manager.cc:194] Registered new tserver with Master: 85dc5621eebb4213b8ee68d8a12698c8 (127.2.74.194:38447)
I20260925 22:41:17.318137 7934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.194@KRBTEST.COM'} at 127.2.74.194:36341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
I20260925 22:41:17.326648 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:0
--local_ip_for_outbound_sockets=127.2.74.195
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:40503
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:45565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:17.448850 8264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:17.449144 8264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:17.449189 8264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:17.453524 8264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:17.453599 8264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:17.453662 8264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:41:17.459609 8264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:45565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:40503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:17.460717 8264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:17.461814 8264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:17.471851 8272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:17.471880 8269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:17.471880 8270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:17.472062 8264 server_base.cc:1034] running on GCE node
I20260925 22:41:17.472513 8264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:17.473206 8264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:17.474406 8264 hybrid_clock.cc:648] HybridClock initialized: now 1790376077474382 us; error 38 us; skew 500 ppm
Sep 25 22:41:17 dist-test-slave-csnl krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376077, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:17.477275 8264 init.cc:377] Logged in from keytab as kudu/127.2.74.195@KRBTEST.COM (short username kudu)
I20260925 22:41:17.478240 8264 webserver.cc:460] Webserver started at http://127.2.74.195:44249/ using document root <none> and password file <none>
I20260925 22:41:17.478823 8264 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:17.478926 8264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:17.479189 8264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:17.481264 8264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-2/data/instance:
uuid: "e61b2db719054329a5ac6469723c335f"
format_stamp: "Formatted at 2026-09-25 22:41:17 on dist-test-slave-csnl"
server_key: "bbb242b806204e76e8915f28ee41b493"
server_key_iv: "f1c3cd8afdd99c51d6ebc2c57e8a6a95"
server_key_version: "encryptionkey@0"
I20260925 22:41:17.481810 8264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance:
uuid: "e61b2db719054329a5ac6469723c335f"
format_stamp: "Formatted at 2026-09-25 22:41:17 on dist-test-slave-csnl"
server_key: "bbb242b806204e76e8915f28ee41b493"
server_key_iv: "f1c3cd8afdd99c51d6ebc2c57e8a6a95"
server_key_version: "encryptionkey@0"
I20260925 22:41:17.485783 8264 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 22:41:17.488327 8279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:17.489650 8264 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:41:17.489799 8264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "e61b2db719054329a5ac6469723c335f"
format_stamp: "Formatted at 2026-09-25 22:41:17 on dist-test-slave-csnl"
server_key: "bbb242b806204e76e8915f28ee41b493"
server_key_iv: "f1c3cd8afdd99c51d6ebc2c57e8a6a95"
server_key_version: "encryptionkey@0"
I20260925 22:41:17.489930 8264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:17.506417 8264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:17.510205 8264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:17.510439 8264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:17.511188 8264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:17.512291 8264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:17.512374 8264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:17.512440 8264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:17.512496 8264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:17.522400 8264 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:36829
I20260925 22:41:17.522434 8392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:36829 every 8 connection(s)
I20260925 22:41:17.523573 8264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:41:17.523902 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 8264
I20260925 22:41:17.524045 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance
I20260925 22:41:17.524447 2347 external_mini_cluster.cc:1428] Setting key 919868922c0a645cc2bb7502c46b9eb9
Sep 25 22:41:17 dist-test-slave-csnl krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376077, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:41:17.538642 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:17.525684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:54359 (local address 127.2.74.254:40503)
0925 22:41:17.525839 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:17.525843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:17.526940 (+ 1097us) server_negotiation.cc:407] Connection header received
0925 22:41:17.528448 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:17.528451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:17.528498 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:17.528576 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:17.529859 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:17.530412 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:17.531761 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:17.531992 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:17.532082 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:17.534689 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:17.534693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:17.534695 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:17.534723 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:17.536338 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:17.536876 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:17.536878 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:17.536880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:17.536976 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:17.537379 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:17.537382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:17.537384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:17.537607 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:17.537733 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:17.538383 (+ 650us) server_negotiation.cc:299] Negotiation successful
0925 22:41:17.538518 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:41:17.539644 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:17.526272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:40503 (local address 127.2.74.195:54359)
0925 22:41:17.526771 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:17.526828 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:17.528191 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:17.528745 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:17.528753 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:17.529155 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:17.529708 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:17.529722 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:17.530540 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:17.530543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:17.531630 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:17.531640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:17.532169 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:17.532172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:17.532336 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:17.532923 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:17.532941 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:17.534504 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:17.536463 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:17.536469 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:17.536494 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:17.536784 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:17.537074 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:17.537077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:17.537079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:17.537279 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:17.537737 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:17.537742 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:17.538060 (+ 318us) client_negotiation.cc:769] Sending connection context
0925 22:41:17.538270 (+ 210us) client_negotiation.cc:240] Negotiation successful
0925 22:41:17.538640 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":156,"threads_started":1}
I20260925 22:41:17.540802 8393 heartbeater.cc:347] Connected to a master server at 127.2.74.254:40503
I20260925 22:41:17.541087 8393 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:17.541672 8393 heartbeater.cc:511] Master 127.2.74.254:40503 requested a full tablet report, sending...
I20260925 22:41:17.542841 7934 ts_manager.cc:194] Registered new tserver with Master: e61b2db719054329a5ac6469723c335f (127.2.74.195:36829)
I20260925 22:41:17.543526 7934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.195@KRBTEST.COM'} at 127.2.74.195:54359
I20260925 22:41:17.551244 2347 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/test-admin.keytab.
Sep 25 22:41:17 dist-test-slave-csnl krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:17.568424 2347 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 22:41:17.571583 8400 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:17.569888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52370 (local address 127.2.74.193:45007)
0925 22:41:17.570235 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:17.570253 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:17.570279 (+ 26us) server_negotiation.cc:407] Connection header received
0925 22:41:17.570340 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:17.570345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:17.570540 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:17.570705 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:17.571361 (+ 656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52370: BlockingRecv error: recv got EOF from 127.0.0.1:52370 (error 108)
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":132,"threads_started":1}
W20260925 22:41:17.571691 2347 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/krb5cc' of type FILE
Sep 25 22:41:17 dist-test-slave-csnl krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:17.572813 2347 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 22:41:17 dist-test-slave-csnl krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:17.573782 2347 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 22:41:17 dist-test-slave-csnl krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.193@KRBTEST.COM
I20260925 22:41:17.584260 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:17.574759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52380 (local address 127.2.74.193:45007)
0925 22:41:17.574908 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:17.574915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:17.575086 (+ 171us) server_negotiation.cc:407] Connection header received
0925 22:41:17.575184 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:17.575186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:17.575230 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:17.575301 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:17.575855 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:17.576652 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:17.577489 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:17.577876 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:17.577968 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:17.580471 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:17.580482 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:17.580498 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:17.580528 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:17.582365 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:17.582812 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:17.582817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:17.582819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:17.582915 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:17.583252 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:17.583254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:17.583255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:17.583542 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:17.583710 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:17.583871 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 22:41:17.584056 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 22:41:17.588326 8400 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:17.587320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52390 (local address 127.2.74.193:45007)
0925 22:41:17.587498 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:17.587501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:17.587558 (+ 57us) server_negotiation.cc:407] Connection header received
0925 22:41:17.587669 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:17.587671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:17.587713 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:17.587780 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:17.588199 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52390: BlockingRecv error: recv got EOF from 127.0.0.1:52390 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260925 22:41:17.588380 2347 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/krb5cc' of type FILE
Sep 25 22:41:17 dist-test-slave-csnl krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:17.589466 2347 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 22:41:17 dist-test-slave-csnl krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:17.590379 2347 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 22:41:17 dist-test-slave-csnl krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.193@KRBTEST.COM
I20260925 22:41:17.599510 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:17.591398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52392 (local address 127.2.74.193:45007)
0925 22:41:17.591545 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:17.591549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:17.591564 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:41:17.591684 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:17.591686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:17.591728 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:17.591798 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:17.592316 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:17.592834 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:17.593701 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:17.593939 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:17.594033 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:17.596585 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:17.596589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:17.596591 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:17.596618 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:17.597909 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:17.598323 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:17.598326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:17.598327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:17.598400 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:17.598624 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:17.598627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:17.598628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:17.598788 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:17.598885 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:17.599176 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 22:41:17.599279 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 22:41:17.602497 8400 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:17.601365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52402 (local address 127.2.74.193:45007)
0925 22:41:17.601518 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:17.601522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:17.601641 (+ 119us) server_negotiation.cc:407] Connection header received
0925 22:41:17.601794 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:17.601797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:17.601839 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:17.601937 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:17.602349 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52402: BlockingRecv error: recv got EOF from 127.0.0.1:52402 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260925 22:41:17.602630 2347 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376047580784-2347-0/krb5kdc/krb5cc' of type FILE
Sep 25 22:41:17 dist-test-slave-csnl krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:17.603734 2347 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 22:41:17 dist-test-slave-csnl krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:17.604624 2347 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 22:41:17 dist-test-slave-csnl krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.193@KRBTEST.COM
I20260925 22:41:17.614187 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:17.605542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52410 (local address 127.2.74.193:45007)
0925 22:41:17.605662 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:17.605666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:17.605860 (+ 194us) server_negotiation.cc:407] Connection header received
0925 22:41:17.606000 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:17.606003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:17.606044 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:17.606148 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:17.606748 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:17.607329 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:17.608163 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:17.608373 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:17.608469 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:17.610798 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:17.610802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:17.610804 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:17.610832 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:17.612407 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:17.612905 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:17.612911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:17.612913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:17.613000 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:17.613289 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:17.613294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:17.613296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:17.613492 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:17.613636 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:17.613897 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 22:41:17.614015 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:41:17.615257 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 7996
I20260925 22:41:17.620708 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 8130
I20260925 22:41:17.626305 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 8264
I20260925 22:41:17.631875 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 7902
2026-09-25T22:41:17Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2203 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:41:17 dist-test-slave-csnl krb5kdc[8413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:41:17 dist-test-slave-csnl krb5kdc[8413](info): set up 2 sockets
Sep 25 22:41:17 dist-test-slave-csnl krb5kdc[8413](info): commencing operation
krb5kdc: starting...
W20260925 22:41:18.687443 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:41:18 dist-test-slave-csnl krb5kdc[8413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:41:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:41:18Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.254.keytab.
Entry for principal oryx/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.254.keytab.
I20260925 22:41:18.814090 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:33523
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.254.keytab
--principal=oryx/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:33523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:18.932822 8426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:18.933149 8426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:18.933192 8426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:18.937356 8426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:41:18.937433 8426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:18.937454 8426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:18.937474 8426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:41:18.937489 8426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:41:18.943076 8426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:33523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.254.keytab
--principal=oryx/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:33523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:18.944185 8426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:18.945204 8426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:41:18.954454 8426 server_base.cc:1034] running on GCE node
W20260925 22:41:18.954325 8434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:18.954325 8432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:18.954325 8431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:18.955379 8426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:18.956598 8426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:18.957780 8426 hybrid_clock.cc:648] HybridClock initialized: now 1790376078957745 us; error 55 us; skew 500 ppm
Sep 25 22:41:18 dist-test-slave-csnl krb5kdc[8413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376078, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:18.961198 8426 init.cc:377] Logged in from keytab as oryx/127.2.74.254@KRBTEST.COM (short username oryx)
I20260925 22:41:18.962313 8426 webserver.cc:460] Webserver started at http://127.2.74.254:33941/ using document root <none> and password file <none>
I20260925 22:41:18.963033 8426 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:18.963140 8426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:18.963399 8426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:18.965641 8426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "f44fda985f2248aa820cd857d80c5e3d"
format_stamp: "Formatted at 2026-09-25 22:41:18 on dist-test-slave-csnl"
server_key: "cf8b166435f3008655fe81380a170899"
server_key_iv: "0cbeb7c8f716c9af10a9ed6f24131a91"
server_key_version: "encryptionkey@0"
I20260925 22:41:18.966284 8426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "f44fda985f2248aa820cd857d80c5e3d"
format_stamp: "Formatted at 2026-09-25 22:41:18 on dist-test-slave-csnl"
server_key: "cf8b166435f3008655fe81380a170899"
server_key_iv: "0cbeb7c8f716c9af10a9ed6f24131a91"
server_key_version: "encryptionkey@0"
I20260925 22:41:18.970613 8426 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:41:18.973599 8441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:18.975142 8426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 22:41:18.975324 8426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "f44fda985f2248aa820cd857d80c5e3d"
format_stamp: "Formatted at 2026-09-25 22:41:18 on dist-test-slave-csnl"
server_key: "cf8b166435f3008655fe81380a170899"
server_key_iv: "0cbeb7c8f716c9af10a9ed6f24131a91"
server_key_version: "encryptionkey@0"
I20260925 22:41:18.975471 8426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:18.994827 8426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:18.998484 8426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:18.998731 8426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:19.006788 8426 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:33523
I20260925 22:41:19.007149 8493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:33523 every 8 connection(s)
I20260925 22:41:19.007968 8426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:41:19.011055 8494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:19.011080 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 8426
I20260925 22:41:19.011219 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:41:19.011536 2347 external_mini_cluster.cc:1428] Setting key e5a13c4e1fd92aac7fd4ab12203d22b3
I20260925 22:41:19.019339 8494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d: Bootstrap starting.
I20260925 22:41:19.022459 8494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d: Neither blocks nor log segments found. Creating new log.
Sep 25 22:41:19 dist-test-slave-csnl krb5kdc[8413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM
I20260925 22:41:19.023471 8494 log.cc:824] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:19.026859 8494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d: No bootstrap required, opened a new log
I20260925 22:41:19.029157 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:19.013062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52112 (local address 127.2.74.254:33523)
0925 22:41:19.013578 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:19.013599 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:19.013809 (+ 210us) server_negotiation.cc:407] Connection header received
0925 22:41:19.014842 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:19.014874 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:19.015411 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:19.015845 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:19.016662 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:19.018239 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:19.019422 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:19.020162 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:19.020342 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:19.023455 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:19.023463 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:19.023485 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:19.023528 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:19.025723 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:19.026371 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:19.026376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:19.026383 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:19.026514 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:19.026862 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:19.026865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:19.026867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:19.027446 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:19.027611 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:19.027952 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 22:41:19.028312 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":149,"threads_started":1}
I20260925 22:41:19.031044 8494 raft_consensus.cc:348] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } }
I20260925 22:41:19.031332 8494 raft_consensus.cc:374] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:19.031395 8494 raft_consensus.cc:729] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f44fda985f2248aa820cd857d80c5e3d, State: Initialized, Role: FOLLOWER
I20260925 22:41:19.031949 8494 consensus_queue.cc:260] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } }
I20260925 22:41:19.032090 8494 raft_consensus.cc:388] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:41:19.032179 8494 raft_consensus.cc:482] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:41:19.032287 8494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:19.033327 8494 raft_consensus.cc:504] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } }
I20260925 22:41:19.033751 8494 leader_election.cc:302] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f44fda985f2248aa820cd857d80c5e3d; no voters:
I20260925 22:41:19.034104 8494 leader_election.cc:288] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:41:19.034245 8499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:41:19.034493 8499 raft_consensus.cc:686] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [term 1 LEADER]: Becoming Leader. State: Replica: f44fda985f2248aa820cd857d80c5e3d, State: Running, Role: LEADER
I20260925 22:41:19.034893 8499 consensus_queue.cc:237] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } }
I20260925 22:41:19.035598 8494 sys_catalog.cc:564] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:41:19.036777 8501 sys_catalog.cc:455] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f44fda985f2248aa820cd857d80c5e3d. Latest consensus state: current_term: 1 leader_uuid: "f44fda985f2248aa820cd857d80c5e3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } }
I20260925 22:41:19.036942 8501 sys_catalog.cc:458] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:19.037218 8500 sys_catalog.cc:455] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f44fda985f2248aa820cd857d80c5e3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } }
I20260925 22:41:19.037314 8500 sys_catalog.cc:458] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:19.037343 8508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:41:19.040385 8508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:41:19.046880 8508 catalog_manager.cc:1347] Generated new cluster ID: 0747fe35ece54cc8b064a2929f65b7e5
I20260925 22:41:19.047025 8508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:41:19.061082 8508 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:41:19.062142 8508 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:41:19.066882 8508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d: Generated new TSK 0
I20260925 22:41:19.067866 8508 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.193.keytab.
Entry for principal oryx/127.2.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.193.keytab.
I20260925 22:41:19.102288 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:0
--local_ip_for_outbound_sockets=127.2.74.193
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:33523
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.193.keytab
--principal=oryx/127.2.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:19.221035 8520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:19.221415 8520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:19.221496 8520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:19.225760 8520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:19.225898 8520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:19.226001 8520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:41:19.231652 8520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.193.keytab
--principal=oryx/127.2.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:33523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:19.232839 8520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:19.233959 8520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:41:19.244374 8520 server_base.cc:1034] running on GCE node
W20260925 22:41:19.244354 8525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:19.244310 8528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:19.244320 8526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:19.245148 8520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:19.245837 8520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:19.247043 8520 hybrid_clock.cc:648] HybridClock initialized: now 1790376079247024 us; error 37 us; skew 500 ppm
Sep 25 22:41:19 dist-test-slave-csnl krb5kdc[8413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376079, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:19.249874 8520 init.cc:377] Logged in from keytab as oryx/127.2.74.193@KRBTEST.COM (short username oryx)
I20260925 22:41:19.251106 8520 webserver.cc:460] Webserver started at http://127.2.74.193:42885/ using document root <none> and password file <none>
I20260925 22:41:19.251705 8520 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:19.251803 8520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:19.252027 8520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:19.254158 8520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/data/instance:
uuid: "200b7a862e10443ba89a5d209f4959aa"
format_stamp: "Formatted at 2026-09-25 22:41:19 on dist-test-slave-csnl"
server_key: "cb7c709dc942e9c218aa493d9b713e10"
server_key_iv: "738c83a39ffc984b04e8b686866015bc"
server_key_version: "encryptionkey@0"
I20260925 22:41:19.254771 8520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance:
uuid: "200b7a862e10443ba89a5d209f4959aa"
format_stamp: "Formatted at 2026-09-25 22:41:19 on dist-test-slave-csnl"
server_key: "cb7c709dc942e9c218aa493d9b713e10"
server_key_iv: "738c83a39ffc984b04e8b686866015bc"
server_key_version: "encryptionkey@0"
I20260925 22:41:19.258692 8520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 22:41:19.261433 8535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:19.262844 8520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:41:19.263006 8520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "200b7a862e10443ba89a5d209f4959aa"
format_stamp: "Formatted at 2026-09-25 22:41:19 on dist-test-slave-csnl"
server_key: "cb7c709dc942e9c218aa493d9b713e10"
server_key_iv: "738c83a39ffc984b04e8b686866015bc"
server_key_version: "encryptionkey@0"
I20260925 22:41:19.263136 8520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:19.276250 8520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:19.279695 8520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:19.279925 8520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:19.280663 8520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:19.281615 8520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:19.281665 8520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:19.281737 8520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:19.281769 8520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:19.291625 8520 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:44813
I20260925 22:41:19.291641 8648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:44813 every 8 connection(s)
I20260925 22:41:19.292589 8520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:41:19.299621 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 8520
I20260925 22:41:19.299734 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance
I20260925 22:41:19.300024 2347 external_mini_cluster.cc:1428] Setting key e1565ab7e368c3e832806317b15b143a
Sep 25 22:41:19 dist-test-slave-csnl krb5kdc[8413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376079, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM
I20260925 22:41:19.307696 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:19.294461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:46379 (local address 127.2.74.254:33523)
0925 22:41:19.294643 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:19.294648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:19.295687 (+ 1039us) server_negotiation.cc:407] Connection header received
0925 22:41:19.297042 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:19.297045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:19.297095 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:19.297176 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:19.298432 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:19.299048 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:19.300409 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:19.300716 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:19.300830 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:19.303721 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:19.303725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:19.303728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:19.303766 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:19.305381 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:19.305964 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:19.305968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:19.305970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:19.306057 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:19.306438 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:19.306441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:19.306443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:19.306650 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:19.306761 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:19.307379 (+ 618us) server_negotiation.cc:299] Negotiation successful
0925 22:41:19.307527 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 22:41:19.308576 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:19.294929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.2.74.193:46379)
0925 22:41:19.295503 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:19.295554 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:19.296823 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:19.297323 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:19.297346 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:19.297734 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:19.298277 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:19.298293 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:19.299171 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:19.299174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:19.300270 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:19.300281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:19.300898 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:19.300903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:19.301102 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:19.301771 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:19.301789 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:19.303515 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:19.305522 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:19.305529 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:19.305551 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:19.305849 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:19.306156 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:19.306161 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:19.306163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:19.306325 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:19.306761 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:19.306769 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:19.307127 (+ 358us) client_negotiation.cc:769] Sending connection context
0925 22:41:19.307339 (+ 212us) client_negotiation.cc:240] Negotiation successful
0925 22:41:19.307699 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for oryx/127.2.74.194@KRBTEST.COM; defaulting to no policy
I20260925 22:41:19.310065 8649 heartbeater.cc:347] Connected to a master server at 127.2.74.254:33523
I20260925 22:41:19.310348 8649 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:19.310904 8649 heartbeater.cc:511] Master 127.2.74.254:33523 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.74.194@KRBTEST.COM" created.
I20260925 22:41:19.312799 8458 ts_manager.cc:194] Registered new tserver with Master: 200b7a862e10443ba89a5d209f4959aa (127.2.74.193:44813)
I20260925 22:41:19.314023 8458 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.74.193@KRBTEST.COM'} at 127.2.74.193:46379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.194.keytab.
Entry for principal oryx/127.2.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.194.keytab.
I20260925 22:41:19.328277 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:0
--local_ip_for_outbound_sockets=127.2.74.194
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:33523
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.194.keytab
--principal=oryx/127.2.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:19.443099 8654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:19.443387 8654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:19.443432 8654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:19.447613 8654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:19.447682 8654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:19.447744 8654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:41:19.453329 8654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.194.keytab
--principal=oryx/127.2.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:33523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:19.454546 8654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:19.455651 8654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:19.465364 8662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:19.465488 8660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:19.465427 8659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:19.465577 8654 server_base.cc:1034] running on GCE node
I20260925 22:41:19.466169 8654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:19.466846 8654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:19.468011 8654 hybrid_clock.cc:648] HybridClock initialized: now 1790376079467958 us; error 63 us; skew 500 ppm
Sep 25 22:41:19 dist-test-slave-csnl krb5kdc[8413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376079, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:19.470783 8654 init.cc:377] Logged in from keytab as oryx/127.2.74.194@KRBTEST.COM (short username oryx)
I20260925 22:41:19.471782 8654 webserver.cc:460] Webserver started at http://127.2.74.194:40505/ using document root <none> and password file <none>
I20260925 22:41:19.472332 8654 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:19.472393 8654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:19.472563 8654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:19.474725 8654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/data/instance:
uuid: "70289c55941042a9969b16ac4d442f7a"
format_stamp: "Formatted at 2026-09-25 22:41:19 on dist-test-slave-csnl"
server_key: "414e83484c09e3ca1c3f0d8889986895"
server_key_iv: "752ca04abc278b1c554c1ba246b1843b"
server_key_version: "encryptionkey@0"
I20260925 22:41:19.475391 8654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance:
uuid: "70289c55941042a9969b16ac4d442f7a"
format_stamp: "Formatted at 2026-09-25 22:41:19 on dist-test-slave-csnl"
server_key: "414e83484c09e3ca1c3f0d8889986895"
server_key_iv: "752ca04abc278b1c554c1ba246b1843b"
server_key_version: "encryptionkey@0"
I20260925 22:41:19.479395 8654 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:41:19.482102 8669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:19.483662 8654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:41:19.483824 8654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "70289c55941042a9969b16ac4d442f7a"
format_stamp: "Formatted at 2026-09-25 22:41:19 on dist-test-slave-csnl"
server_key: "414e83484c09e3ca1c3f0d8889986895"
server_key_iv: "752ca04abc278b1c554c1ba246b1843b"
server_key_version: "encryptionkey@0"
I20260925 22:41:19.483960 8654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:19.503648 8654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:19.507297 8654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:19.507529 8654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:19.508152 8654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:19.509021 8654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:19.509093 8654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:19.509168 8654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:19.509212 8654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:19.518381 8654 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:38067
I20260925 22:41:19.518407 8782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:38067 every 8 connection(s)
I20260925 22:41:19.519376 8654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:41:19.525067 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 8654
I20260925 22:41:19.525204 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance
I20260925 22:41:19.525528 2347 external_mini_cluster.cc:1428] Setting key 6b64a9626623c9e0361527a2a3b242bf
Sep 25 22:41:19 dist-test-slave-csnl krb5kdc[8413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376079, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM
I20260925 22:41:19.534850 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:19.521309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:47527 (local address 127.2.74.254:33523)
0925 22:41:19.521443 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:19.521446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:19.522290 (+ 844us) server_negotiation.cc:407] Connection header received
0925 22:41:19.523744 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:19.523748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:19.523812 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:19.523894 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:19.525358 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:19.526086 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:19.527589 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:19.527812 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:19.527927 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:19.531166 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:19.531169 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:19.531171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:19.531196 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:19.532656 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:19.533221 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:19.533225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:19.533226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:19.533299 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:19.533663 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:19.533666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:19.533668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:19.533917 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:19.534026 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:19.534554 (+ 528us) server_negotiation.cc:299] Negotiation successful
0925 22:41:19.534681 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for oryx/127.2.74.195@KRBTEST.COM; defaulting to no policy
I20260925 22:41:19.535822 8785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:19.521665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.2.74.194:47527)
0925 22:41:19.522124 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:19.522171 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:19.523500 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:19.524077 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:19.524086 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:19.524471 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:19.525153 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:19.525179 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:19.526223 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:19.526227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:19.527452 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:19.527462 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:19.527978 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:19.527983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:19.528185 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:19.529067 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:19.529097 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:19.530910 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:19.532786 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:19.532793 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:19.532811 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:19.533101 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:19.533404 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:19.533407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:19.533409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:19.533561 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:19.534029 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:19.534039 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:19.534302 (+ 263us) client_negotiation.cc:769] Sending connection context
0925 22:41:19.534491 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 22:41:19.534841 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.74.195@KRBTEST.COM" created.
I20260925 22:41:19.537361 8783 heartbeater.cc:347] Connected to a master server at 127.2.74.254:33523
I20260925 22:41:19.537633 8783 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:19.538287 8783 heartbeater.cc:511] Master 127.2.74.254:33523 requested a full tablet report, sending...
I20260925 22:41:19.539521 8458 ts_manager.cc:194] Registered new tserver with Master: 70289c55941042a9969b16ac4d442f7a (127.2.74.194:38067)
I20260925 22:41:19.540136 8458 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.74.194@KRBTEST.COM'} at 127.2.74.194:47527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.195.keytab.
Entry for principal oryx/127.2.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.195.keytab.
I20260925 22:41:19.554195 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:0
--local_ip_for_outbound_sockets=127.2.74.195
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:33523
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.195.keytab
--principal=oryx/127.2.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:19.669255 8788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:19.669554 8788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:19.669600 8788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:19.673674 8788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:19.673748 8788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:19.673812 8788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:41:19.679261 8788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.195.keytab
--principal=oryx/127.2.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:33523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:19.680518 8788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:19.681602 8788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:19.692214 8794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:19.692430 8793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:19.692526 8796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:19.692564 8788 server_base.cc:1034] running on GCE node
I20260925 22:41:19.693081 8788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:19.693835 8788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:19.695036 8788 hybrid_clock.cc:648] HybridClock initialized: now 1790376079695013 us; error 36 us; skew 500 ppm
Sep 25 22:41:19 dist-test-slave-csnl krb5kdc[8413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376079, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:19.698024 8788 init.cc:377] Logged in from keytab as oryx/127.2.74.195@KRBTEST.COM (short username oryx)
I20260925 22:41:19.699149 8788 webserver.cc:460] Webserver started at http://127.2.74.195:39241/ using document root <none> and password file <none>
I20260925 22:41:19.699769 8788 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:19.699868 8788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:19.700115 8788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:19.702319 8788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/data/instance:
uuid: "e74fb20181524e049aca671bdc2d4bbd"
format_stamp: "Formatted at 2026-09-25 22:41:19 on dist-test-slave-csnl"
server_key: "ca5b5d64dfa2a0835e7cd825641ef53f"
server_key_iv: "d523020d106d3670f3e0c7674e9a92aa"
server_key_version: "encryptionkey@0"
I20260925 22:41:19.703022 8788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance:
uuid: "e74fb20181524e049aca671bdc2d4bbd"
format_stamp: "Formatted at 2026-09-25 22:41:19 on dist-test-slave-csnl"
server_key: "ca5b5d64dfa2a0835e7cd825641ef53f"
server_key_iv: "d523020d106d3670f3e0c7674e9a92aa"
server_key_version: "encryptionkey@0"
I20260925 22:41:19.707294 8788 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:41:19.710495 8803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:19.712062 8788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:41:19.712191 8788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "e74fb20181524e049aca671bdc2d4bbd"
format_stamp: "Formatted at 2026-09-25 22:41:19 on dist-test-slave-csnl"
server_key: "ca5b5d64dfa2a0835e7cd825641ef53f"
server_key_iv: "d523020d106d3670f3e0c7674e9a92aa"
server_key_version: "encryptionkey@0"
I20260925 22:41:19.712302 8788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:19.730587 8788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:19.733866 8788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:19.734040 8788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:19.734620 8788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:19.735653 8788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:19.735702 8788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:19.735778 8788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:19.735819 8788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:19.745348 8788 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:37181
I20260925 22:41:19.745395 8916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:37181 every 8 connection(s)
I20260925 22:41:19.746349 8788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:41:19.751194 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 8788
I20260925 22:41:19.751360 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance
I20260925 22:41:19.751688 2347 external_mini_cluster.cc:1428] Setting key e071774ef5888aa97456f20f4e34df15
Sep 25 22:41:19 dist-test-slave-csnl krb5kdc[8413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376079, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM
I20260925 22:41:19.760758 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:19.748322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:45791 (local address 127.2.74.254:33523)
0925 22:41:19.748459 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:19.748463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:19.749395 (+ 932us) server_negotiation.cc:407] Connection header received
0925 22:41:19.750807 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:19.750811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:19.750869 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:19.750971 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:19.752533 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:19.753176 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:19.754367 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:19.754595 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:19.754711 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:19.757284 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:19.757287 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:19.757289 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:19.757315 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:19.758592 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:19.759210 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:19.759218 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:19.759219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:19.759292 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:19.759645 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:19.759648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:19.759650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:19.759838 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:19.759945 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:19.760470 (+ 525us) server_negotiation.cc:299] Negotiation successful
0925 22:41:19.760592 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:41:19.761622 8919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:19.748753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.2.74.195:45791)
0925 22:41:19.749230 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:19.749278 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:19.750574 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:19.751167 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:19.751175 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:19.751732 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:19.752377 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:19.752398 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:19.753312 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:19.753316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:19.754237 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:19.754247 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:19.754754 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:19.754758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:19.755014 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:19.755603 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:19.755620 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:19.757093 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:19.758718 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:19.758724 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:19.758742 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:19.759099 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:19.759394 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:19.759397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:19.759399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:19.759544 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:19.759943 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:19.759960 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:19.760222 (+ 262us) client_negotiation.cc:769] Sending connection context
0925 22:41:19.760412 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 22:41:19.760775 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":116,"threads_started":1}
I20260925 22:41:19.762864 8917 heartbeater.cc:347] Connected to a master server at 127.2.74.254:33523
I20260925 22:41:19.763144 8917 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:19.763675 8917 heartbeater.cc:511] Master 127.2.74.254:33523 requested a full tablet report, sending...
I20260925 22:41:19.764837 8457 ts_manager.cc:194] Registered new tserver with Master: e74fb20181524e049aca671bdc2d4bbd (127.2.74.195:37181)
I20260925 22:41:19.765558 8457 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.74.195@KRBTEST.COM'} at 127.2.74.195:45791
I20260925 22:41:19.765688 2347 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.74.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.74.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.74.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.253.keytab.
Entry for principal oryx/127.2.74.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.253.keytab.
I20260925 22:41:19.792411 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.253:44125
--webserver_interface=127.2.74.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.253.keytab
--principal=oryx/127.2.74.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:33523,127.2.74.253:44125
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:19.909570 8922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:19.909895 8922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:19.909981 8922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:19.914366 8922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:41:19.914464 8922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:19.914485 8922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:19.914503 8922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:41:19.914518 8922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:41:19.920512 8922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:33523,127.2.74.253:44125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.253.keytab
--principal=oryx/127.2.74.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.253:44125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.74.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
W20260925 22:41:19.920801 8922 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.74.254:33523,127.2.74.253:44125'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 22:41:19.921801 8922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:19.922921 8922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:19.932123 8928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:19.932137 8930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:19.932164 8927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:19.932336 8922 server_base.cc:1034] running on GCE node
I20260925 22:41:19.932969 8922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:19.934233 8922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:19.935407 8922 hybrid_clock.cc:648] HybridClock initialized: now 1790376079935384 us; error 44 us; skew 500 ppm
Sep 25 22:41:19 dist-test-slave-csnl krb5kdc[8413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376079, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:19.938635 8922 init.cc:377] Logged in from keytab as oryx/127.2.74.253@KRBTEST.COM (short username oryx)
I20260925 22:41:19.939706 8922 webserver.cc:460] Webserver started at http://127.2.74.253:37651/ using document root <none> and password file <none>
I20260925 22:41:19.940320 8922 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:19.940390 8922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:19.940670 8922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:19.942888 8922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-1/data/instance:
uuid: "fbdb039a1ac44df999d6b05de7cb0f77"
format_stamp: "Formatted at 2026-09-25 22:41:19 on dist-test-slave-csnl"
server_key: "9c9a99246f011992b020d43281dfd180"
server_key_iv: "d09f05e321224f54b2ad37d3466d9a3e"
server_key_version: "encryptionkey@0"
I20260925 22:41:19.943601 8922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-1/wal/instance:
uuid: "fbdb039a1ac44df999d6b05de7cb0f77"
format_stamp: "Formatted at 2026-09-25 22:41:19 on dist-test-slave-csnl"
server_key: "9c9a99246f011992b020d43281dfd180"
server_key_iv: "d09f05e321224f54b2ad37d3466d9a3e"
server_key_version: "encryptionkey@0"
I20260925 22:41:19.947667 8922 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:41:19.950366 8937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:19.951723 8922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:41:19.951879 8922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-1/wal
uuid: "fbdb039a1ac44df999d6b05de7cb0f77"
format_stamp: "Formatted at 2026-09-25 22:41:19 on dist-test-slave-csnl"
server_key: "9c9a99246f011992b020d43281dfd180"
server_key_iv: "d09f05e321224f54b2ad37d3466d9a3e"
server_key_version: "encryptionkey@0"
I20260925 22:41:19.952008 8922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:19.968779 8922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:19.971982 8922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:19.972225 8922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:19.979820 8922 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.253:44125
I20260925 22:41:19.980135 8989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.253:44125 every 8 connection(s)
I20260925 22:41:19.981024 8922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-1/data/info.pb
I20260925 22:41:19.984210 8990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:19.988595 8990 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:41:19.989527 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 8922
Sep 25 22:41:19 dist-test-slave-csnl krb5kdc[8413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376079, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.253@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM
Sep 25 22:41:19 dist-test-slave-csnl krb5kdc[8413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.74.253@KRBTEST.COM
I20260925 22:41:20.003634 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:19.990874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52114 (local address 127.2.74.254:33523)
0925 22:41:19.991030 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:19.991035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:19.991756 (+ 721us) server_negotiation.cc:407] Connection header received
0925 22:41:19.992612 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:19.992615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:19.992661 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:19.992757 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:19.993886 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:19.994689 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:19.996249 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:19.996584 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:19.996694 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:19.999712 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:19.999716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:19.999718 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:19.999743 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:20.001390 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.001997 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.002000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.002002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.002085 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.002437 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.002441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.002442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.002677 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:20.002787 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:20.003329 (+ 542us) server_negotiation.cc:299] Negotiation successful
0925 22:41:20.003458 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:41:20.004525 8993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:19.991085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.0.0.1:52114)
0925 22:41:19.991595 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:19.991644 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:19.992394 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:19.992852 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:19.992858 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:19.993358 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:19.993748 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:19.993761 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:19.994827 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:19.994830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:19.996129 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:19.996141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:19.996861 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:19.996868 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:19.996991 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:19.997685 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:19.997708 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:19.999462 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:20.001505 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.001522 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.001534 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.001914 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.002179 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.002184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.002186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.002333 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.002782 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:20.002789 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:20.003102 (+ 313us) client_negotiation.cc:769] Sending connection context
0925 22:41:20.003272 (+ 170us) client_negotiation.cc:240] Negotiation successful
0925 22:41:20.003611 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":170,"threads_started":1}
I20260925 22:41:20.004523 8994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:19.991088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50708 (local address 127.2.74.253:44125)
0925 22:41:19.991613 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:19.991627 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:19.991659 (+ 32us) server_negotiation.cc:407] Connection header received
0925 22:41:19.992372 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:19.992389 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:19.992642 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:19.992818 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:19.993717 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:19.995043 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:19.996177 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:19.996759 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:19.996890 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.000060 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:20.000066 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:20.000078 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:20.000098 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:20.001813 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.002253 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.002259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.002260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.002350 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.002637 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.002641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.002643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.003116 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:20.003327 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:20.003558 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 22:41:20.003703 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":411,"spinlock_wait_cycles":25344,"thread_start_us":173,"threads_started":1}
I20260925 22:41:20.006376 8990 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:41:20 dist-test-slave-csnl krb5kdc[8413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376079, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.253@KRBTEST.COM for oryx/127.2.74.253@KRBTEST.COM
I20260925 22:41:20.019819 8994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.007279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50720 (local address 127.2.74.253:44125)
0925 22:41:20.007389 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:20.007396 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:20.011303 (+ 3907us) server_negotiation.cc:407] Connection header received
0925 22:41:20.011508 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:20.011511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:20.011558 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:20.011654 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:20.012123 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.012655 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.013421 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.013660 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.013774 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.016018 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:20.016021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:20.016023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:20.016044 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:20.018089 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.018558 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.018564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.018566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.018643 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.018922 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.018926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.018929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.019152 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:20.019262 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:20.019513 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 22:41:20.019686 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":29}
I20260925 22:41:20.019819 8993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.007052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:44125 (local address 127.0.0.1:50720)
0925 22:41:20.011179 (+ 4127us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:20.011198 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:20.011393 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:20.011669 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:20.011672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:20.011891 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:20.011997 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.012006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.012767 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.012773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.013294 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.013301 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.013779 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.013782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.013867 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.014437 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:20.014449 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:20.015852 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:20.018238 (+ 2386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.018241 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.018244 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.018457 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.018743 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.018746 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.018747 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.018830 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.019250 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:20.019253 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:20.019335 (+ 82us) client_negotiation.cc:769] Sending connection context
0925 22:41:20.019460 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 22:41:20.019622 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4013}
I20260925 22:41:20.023061 8990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77: Bootstrap starting.
I20260925 22:41:20.025482 8990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:20.026197 8990 log.cc:824] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:20.028910 8990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77: No bootstrap required, opened a new log
I20260925 22:41:20.032281 8990 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } }
I20260925 22:41:20.032542 8990 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:20.032577 8990 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbdb039a1ac44df999d6b05de7cb0f77, State: Initialized, Role: FOLLOWER
I20260925 22:41:20.033048 8990 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } }
I20260925 22:41:20.033576 8996 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } } }
I20260925 22:41:20.033753 8996 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:41:20.034373 8990 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 22:41:20.039901 9007 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:41:20.040026 9007 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:41:20.041882 8922 master_runner.cc:428] Detected that this master fbdb039a1ac44df999d6b05de7cb0f77 is joining an existing cluster
I20260925 22:41:20.041942 8922 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.74.253:44125
I20260925 22:41:20.052433 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.044553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52128 (local address 127.2.74.254:33523)
0925 22:41:20.044680 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:20.044685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:20.044700 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:41:20.044745 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:20.044748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:20.044794 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:20.044883 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:20.045364 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.046203 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.047174 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.047418 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.047533 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.048784 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:20.048788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:20.048790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:20.048816 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:20.050519 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.051073 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.051077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.051078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.051168 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.051447 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.051451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.051453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.051697 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:20.051799 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:20.052142 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 22:41:20.052271 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:41:20.052596 8994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.044577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50732 (local address 127.2.74.253:44125)
0925 22:41:20.044685 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:20.044691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:20.045213 (+ 522us) server_negotiation.cc:407] Connection header received
0925 22:41:20.045302 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:20.045308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:20.045363 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:20.045432 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:20.045816 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.046366 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.047533 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.047856 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.047927 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.049175 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:20.049182 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:20.049186 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:20.049214 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:20.050814 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.051349 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.051353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.051354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.051422 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.051675 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.051678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.051680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.051947 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:20.052042 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:20.052308 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 22:41:20.052417 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":37}
I20260925 22:41:20.052429 9013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.044166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.0.0.1:52128)
0925 22:41:20.044486 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:20.044502 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:20.044613 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:20.044892 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:20.044894 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:20.045083 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:20.045235 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.045242 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.046323 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.046327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.047055 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.047064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.047535 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.047539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.047630 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.048185 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:20.048197 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:20.048648 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:20.050639 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.050643 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.050649 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.050935 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.051270 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.051273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.051275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.051360 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.051798 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:20.051801 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:20.051936 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 22:41:20.052036 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 22:41:20.052197 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":22,"thread_start_us":115,"threads_started":1}
I20260925 22:41:20.052851 9014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.044203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:44125 (local address 127.0.0.1:50732)
0925 22:41:20.045111 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:20.045124 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:20.045218 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:20.045457 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:20.045459 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:20.045623 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:20.045708 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.045713 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.046492 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.046495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.047403 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.047414 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.048083 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.048087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.048177 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.048616 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:20.048645 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:20.049054 (+ 409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:20.050930 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.050933 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.050936 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.051265 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.051500 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.051503 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.051505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.051581 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.052074 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:20.052077 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:20.052154 (+ 77us) client_negotiation.cc:769] Sending connection context
0925 22:41:20.052472 (+ 318us) client_negotiation.cc:240] Negotiation successful
0925 22:41:20.052582 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"spinlock_wait_cycles":896,"thread_start_us":125,"threads_started":1}
Sep 25 22:41:20 dist-test-slave-csnl krb5kdc[8413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376078, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.254@KRBTEST.COM for oryx/127.2.74.253@KRBTEST.COM
WARNING: no policy specified for oryx/127.2.74.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.74.252@KRBTEST.COM" created.
I20260925 22:41:20.070910 8994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.059684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50734 (local address 127.2.74.253:44125)
0925 22:41:20.059822 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:20.059837 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:20.059853 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:41:20.059928 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:20.059934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:20.060000 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:20.060085 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:20.060980 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.061571 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.063055 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.063475 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.063571 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.066937 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:20.066941 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:20.066961 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:20.066994 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:20.068815 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.069454 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.069459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.069461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.069543 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.069833 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.069837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.069841 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.070106 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:20.070203 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:20.070614 (+ 411us) server_negotiation.cc:299] Negotiation successful
0925 22:41:20.070740 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 22:41:20.070984 9015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.058884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:44125 (local address 127.0.0.1:50734)
0925 22:41:20.059347 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:20.059412 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:20.059556 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:20.060281 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:20.060288 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:20.060706 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:20.060836 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.060843 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.061715 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.061722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.062866 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.062883 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.063769 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.063776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.063941 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.064681 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:20.064703 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:20.066685 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:20.068943 (+ 2258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.068953 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.068960 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.069342 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.069646 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.069649 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.069651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.069740 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.070203 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:20.070213 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:20.070350 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 22:41:20.070565 (+ 215us) client_negotiation.cc:240] Negotiation successful
0925 22:41:20.070783 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":134,"threads_started":1}
I20260925 22:41:20.072690 8457 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.74.253:44125: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77"
member_type: NON_VOTER
last_known_addr {
host: "127.2.74.253"
port: 44125
}
attrs {
promote: true
}
}
dest_uuid: "f44fda985f2248aa820cd857d80c5e3d"
cas_config_opid_index: -1
I20260925 22:41:20.073503 8457 consensus_queue.cc:237] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: NON_VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: true } }
I20260925 22:41:20.075757 9019 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER fbdb039a1ac44df999d6b05de7cb0f77 (127.2.74.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: NON_VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: true } } }
I20260925 22:41:20.076839 8964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:20.077847 9019 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.74.253:44125
I20260925 22:41:20.078339 8922 master.cc:537] Master@127.2.74.253:44125 shutting down...
I20260925 22:41:20.078524 8964 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [term 1 FOLLOWER]: Refusing update from remote peer f44fda985f2248aa820cd857d80c5e3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 22:41:20.080646 9019 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: NON_VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:41:20.081349 9020 sys_catalog.cc:455] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f44fda985f2248aa820cd857d80c5e3d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: NON_VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: true } } }
I20260925 22:41:20.081492 9020 sys_catalog.cc:458] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [sys.catalog]: This master's current role is: LEADER
W20260925 22:41:20.083900 8442 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260925 22:41:20.084161 8922 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:41:20.084471 8922 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:41:20.084698 8922 tablet_replica.cc:330] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77: stopping tablet replica
W20260925 22:41:20.084906 8442 consensus_peers.cc:487] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d -> Peer fbdb039a1ac44df999d6b05de7cb0f77 (127.2.74.253:44125): Couldn't send request to peer fbdb039a1ac44df999d6b05de7cb0f77. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.74.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.252.keytab.
Entry for principal oryx/127.2.74.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.252.keytab.
I20260925 22:41:20.089663 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.252:43589
--webserver_interface=127.2.74.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.252.keytab
--principal=oryx/127.2.74.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:33523,127.2.74.253:44125,127.2.74.252:43589
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:41:20.089924 8922 master.cc:552] Master@127.2.74.253:44125 shutdown complete.
I20260925 22:41:20.090003 8922 master_runner.cc:305] Clearing existing system tablet
I20260925 22:41:20.090286 8922 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:20.090354 8922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:20.091854 8922 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 22:41:20.092864 9026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:20.093088 8922 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:20.093266 8922 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:41:20.093309 8922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-1/wal
uuid: "fbdb039a1ac44df999d6b05de7cb0f77"
format_stamp: "Formatted at 2026-09-25 22:41:19 on dist-test-slave-csnl"
server_key: "9c9a99246f011992b020d43281dfd180"
server_key_iv: "d09f05e321224f54b2ad37d3466d9a3e"
server_key_version: "encryptionkey@0"
I20260925 22:41:20.096321 8922 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:41:20.099587 8922 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 22:41:20.099712 8922 log.cc:1196] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 22:41:20.100090 8922 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77: Deleting consensus metadata
I20260925 22:41:20.100301 8922 master_runner.cc:315] Copying system tablet from 127.2.74.254:33523
I20260925 22:41:20.101792 8922 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77: tablet copy: Beginning tablet copy session from remote peer at address 127.2.74.254:33523
I20260925 22:41:20.111434 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.102606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52142 (local address 127.2.74.254:33523)
0925 22:41:20.102765 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:20.102772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:20.102898 (+ 126us) server_negotiation.cc:407] Connection header received
0925 22:41:20.103107 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:20.103110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:20.103165 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:20.103267 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:20.103799 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.104642 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.105848 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.106206 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.106310 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.107693 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:20.107697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:20.107700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:20.107727 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:20.109536 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.110062 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.110066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.110068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.110162 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.110463 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.110466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.110468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.110692 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:20.110787 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:20.111133 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 22:41:20.111264 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:41:20.111454 9032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.102494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.0.0.1:52142)
0925 22:41:20.102799 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:20.102813 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:20.102929 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:20.103302 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:20.103305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:20.103527 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:20.103671 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.103678 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.104768 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.104772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.105710 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.105722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.106329 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.106333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.106433 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.107076 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:20.107091 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:20.107530 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:20.109667 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.109671 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.109673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.109945 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.110272 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.110275 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.110277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.110370 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.110786 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:20.110788 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:20.110894 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 22:41:20.111071 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 22:41:20.111241 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":87,"threads_started":1}
I20260925 22:41:20.112558 8477 tablet_copy_service.cc:140] P f44fda985f2248aa820cd857d80c5e3d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fbdb039a1ac44df999d6b05de7cb0f77 ({username='oryx', principal='oryx/127.2.74.253@KRBTEST.COM'} at 127.0.0.1:52142)
I20260925 22:41:20.112771 8477 tablet_copy_service.cc:161] P f44fda985f2248aa820cd857d80c5e3d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fbdb039a1ac44df999d6b05de7cb0f77 at {username='oryx', principal='oryx/127.2.74.253@KRBTEST.COM'} at 127.0.0.1:52142: session id = fbdb039a1ac44df999d6b05de7cb0f77-00000000000000000000000000000000
I20260925 22:41:20.114768 8477 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 22:41:20.118160 8922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 22:41:20.120967 8922 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77: tablet copy: Starting download of 0 data blocks...
I20260925 22:41:20.121348 8922 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77: tablet copy: Starting download of 1 WAL segments...
I20260925 22:41:20.123274 8922 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 22:41:20.161940 8477 tablet_copy_service.cc:342] P f44fda985f2248aa820cd857d80c5e3d: Request end of tablet copy session fbdb039a1ac44df999d6b05de7cb0f77-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.74.253@KRBTEST.COM'} at 127.0.0.1:52142
I20260925 22:41:20.162137 8477 tablet_copy_service.cc:433] P f44fda985f2248aa820cd857d80c5e3d: ending tablet copy session fbdb039a1ac44df999d6b05de7cb0f77-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fbdb039a1ac44df999d6b05de7cb0f77
I20260925 22:41:20.171814 8922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:20.174000 9036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:20.174083 9033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:20.174418 9034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:20.175165 8922 server_base.cc:1034] running on GCE node
I20260925 22:41:20.175356 8922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:20.175698 8922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:20.176872 8922 hybrid_clock.cc:648] HybridClock initialized: now 1790376080176839 us; error 49 us; skew 500 ppm
Sep 25 22:41:20 dist-test-slave-csnl krb5kdc[8413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376080, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:20.178817 8922 init.cc:377] Logged in from keytab as oryx/127.2.74.253@KRBTEST.COM (short username oryx)
I20260925 22:41:20.179416 8922 webserver.cc:460] Webserver started at http://127.2.74.253:40941/ using document root <none> and password file <none>
I20260925 22:41:20.179629 8922 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:20.179683 8922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:20.181241 8922 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 22:41:20.182861 9043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:20.183226 8922 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 22:41:20.183341 8922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-1/wal
uuid: "fbdb039a1ac44df999d6b05de7cb0f77"
format_stamp: "Formatted at 2026-09-25 22:41:19 on dist-test-slave-csnl"
server_key: "9c9a99246f011992b020d43281dfd180"
server_key_iv: "d09f05e321224f54b2ad37d3466d9a3e"
server_key_version: "encryptionkey@0"
I20260925 22:41:20.183475 8922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:20.193652 8922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:20.196055 8922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:20.201010 8922 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.253:44125
I20260925 22:41:20.201035 9095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.253:44125 every 8 connection(s)
I20260925 22:41:20.201779 8922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-1/data/info.pb
I20260925 22:41:20.203699 9096 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:41:20.204808 9096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77: Bootstrap starting.
I20260925 22:41:20.216490 9096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:41:20.216867 9096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77: Bootstrap complete.
I20260925 22:41:20.217547 9096 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: NON_VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: true } }
I20260925 22:41:20.217712 9096 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fbdb039a1ac44df999d6b05de7cb0f77, State: Initialized, Role: LEARNER
I20260925 22:41:20.217823 9096 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: NON_VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: true } }
I20260925 22:41:20.218183 9099 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: NON_VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: true } } }
I20260925 22:41:20.218319 9099 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [sys.catalog]: This master's current role is: LEARNER
I20260925 22:41:20.218340 9096 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:41:20.221839 9110 catalog_manager.cc:1259] Loaded cluster ID: 0747fe35ece54cc8b064a2929f65b7e5
I20260925 22:41:20.221904 9110 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77: loading cluster ID for follower catalog manager: success
I20260925 22:41:20.223572 9110 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77: acquiring CA information for follower catalog manager: success
I20260925 22:41:20.224808 9110 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 22:41:20.238237 9024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:20.238595 9024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:20.238680 9024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:20.243204 9024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:41:20.243300 9024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:20.243346 9024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:20.243404 9024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:41:20.243443 9024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:41:20.249248 9024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:33523,127.2.74.253:44125,127.2.74.252:43589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.252.keytab
--principal=oryx/127.2.74.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.252:43589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.74.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:20.250483 9024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:20.251641 9024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:20.261173 9118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:20.261220 9116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:20.261220 9115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:20.261600 9024 server_base.cc:1034] running on GCE node
I20260925 22:41:20.262290 9024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:20.263708 9024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:20.264892 9024 hybrid_clock.cc:648] HybridClock initialized: now 1790376080264877 us; error 30 us; skew 500 ppm
Sep 25 22:41:20 dist-test-slave-csnl krb5kdc[8413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376080, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:20.268163 9024 init.cc:377] Logged in from keytab as oryx/127.2.74.252@KRBTEST.COM (short username oryx)
I20260925 22:41:20.269313 9024 webserver.cc:460] Webserver started at http://127.2.74.252:33843/ using document root <none> and password file <none>
I20260925 22:41:20.269924 9024 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:20.270033 9024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:20.270265 9024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:20.272567 9024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-2/data/instance:
uuid: "8c135ec404cb42e7b809b5b70986d83b"
format_stamp: "Formatted at 2026-09-25 22:41:20 on dist-test-slave-csnl"
server_key: "1545f0f5184ec273ce477860947598ca"
server_key_iv: "54b928ca903ed184d4b8fab26db1d836"
server_key_version: "encryptionkey@0"
I20260925 22:41:20.273269 9024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-2/wal/instance:
uuid: "8c135ec404cb42e7b809b5b70986d83b"
format_stamp: "Formatted at 2026-09-25 22:41:20 on dist-test-slave-csnl"
server_key: "1545f0f5184ec273ce477860947598ca"
server_key_iv: "54b928ca903ed184d4b8fab26db1d836"
server_key_version: "encryptionkey@0"
I20260925 22:41:20.277681 9024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:41:20.280645 9125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:20.282047 9024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:41:20.282222 9024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-2/wal
uuid: "8c135ec404cb42e7b809b5b70986d83b"
format_stamp: "Formatted at 2026-09-25 22:41:20 on dist-test-slave-csnl"
server_key: "1545f0f5184ec273ce477860947598ca"
server_key_iv: "54b928ca903ed184d4b8fab26db1d836"
server_key_version: "encryptionkey@0"
I20260925 22:41:20.282361 9024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:20.297302 9024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:20.300593 9024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:20.300830 9024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:20.308101 9024 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.252:43589
I20260925 22:41:20.308437 9177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.252:43589 every 8 connection(s)
I20260925 22:41:20.309203 9024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-2/data/info.pb
I20260925 22:41:20.312409 9178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:20.316458 9178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:41:20.316805 8649 heartbeater.cc:503] Master 127.2.74.254:33523 was elected leader, sending a full tablet report...
I20260925 22:41:20.318233 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 9024
Sep 25 22:41:20 dist-test-slave-csnl krb5kdc[8413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376080, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.252@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM
Sep 25 22:41:20 dist-test-slave-csnl krb5kdc[8413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.74.252@KRBTEST.COM
I20260925 22:41:20.332854 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.318773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52146 (local address 127.2.74.254:33523)
0925 22:41:20.318986 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:20.318993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:20.320652 (+ 1659us) server_negotiation.cc:407] Connection header received
0925 22:41:20.321270 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:20.321273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:20.321333 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:20.321430 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:20.322612 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.323449 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.324984 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.325322 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.325432 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.328653 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:20.328661 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:20.328664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:20.328702 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:20.330309 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.330890 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.330892 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.330893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.331034 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.331450 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.331454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.331456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.331727 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:20.331813 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:20.332525 (+ 712us) server_negotiation.cc:299] Negotiation successful
0925 22:41:20.332671 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:41:20.333969 9181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.319213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.0.0.1:52146)
0925 22:41:20.320285 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:20.320324 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:20.321039 (+ 715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:20.321541 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:20.321548 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:20.322068 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:20.322469 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.322491 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.323596 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.323599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.324862 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.324875 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.325477 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.325480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.325664 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.326468 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:20.326491 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:20.328417 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:20.330428 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.330438 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.330452 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.330780 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.331138 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.331142 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.331145 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.331352 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.331833 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:20.331841 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:20.332225 (+ 384us) client_negotiation.cc:769] Sending connection context
0925 22:41:20.332453 (+ 228us) client_negotiation.cc:240] Negotiation successful
0925 22:41:20.332835 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"spinlock_wait_cycles":23552,"thread_start_us":193,"threads_started":1}
I20260925 22:41:20.333969 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.319217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36854 (local address 127.2.74.252:43589)
0925 22:41:20.319745 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:20.319765 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:20.319798 (+ 33us) server_negotiation.cc:407] Connection header received
0925 22:41:20.321023 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:20.321047 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:20.321336 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:20.321522 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:20.322477 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.323739 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.324979 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.325579 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.325665 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.329106 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:20.329117 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:20.329136 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:20.329170 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:20.331228 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.331736 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.331742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.331743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.331842 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.332126 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.332129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.332130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.332393 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:20.332588 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:20.332805 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 22:41:20.332928 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"spinlock_wait_cycles":11904,"thread_start_us":191,"threads_started":1}
I20260925 22:41:20.335335 9178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:41:20 dist-test-slave-csnl krb5kdc[8413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376080, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.252@KRBTEST.COM for oryx/127.2.74.253@KRBTEST.COM
I20260925 22:41:20.343832 9181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.335901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:44125 (local address 127.0.0.1:50744)
0925 22:41:20.336012 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:20.336026 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:20.336153 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:20.336739 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:20.336742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:20.336931 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:20.337086 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.337091 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.338078 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.338081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.338627 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.338634 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.339122 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.339125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.339203 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.339668 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:20.339680 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:20.340901 (+ 1221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:20.342435 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.342437 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.342439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.342635 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.342880 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.342882 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.342883 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.343007 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.343392 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:20.343394 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:20.343491 (+ 97us) client_negotiation.cc:769] Sending connection context
0925 22:41:20.343585 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 22:41:20.343703 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 22:41:20.343906 9182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.336219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50744 (local address 127.2.74.253:44125)
0925 22:41:20.336494 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:20.336498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:20.336509 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:41:20.336571 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:20.336574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:20.336628 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:20.336738 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:20.337194 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.337981 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.338730 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.339004 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.339117 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.341045 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:20.341048 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:20.341050 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:20.341080 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:20.342334 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.342723 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.342725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.342726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.342801 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.343089 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.343091 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.343092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.343283 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:20.343363 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:20.343645 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 22:41:20.343751 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":122,"threads_started":1}
I20260925 22:41:20.344677 9178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.252" port: 43589 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:41:20 dist-test-slave-csnl krb5kdc[8413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376080, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.252@KRBTEST.COM for oryx/127.2.74.252@KRBTEST.COM
I20260925 22:41:20.351835 9181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.345169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:43589 (local address 127.0.0.1:36864)
0925 22:41:20.345287 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:20.345302 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:20.345374 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:20.345590 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:20.345592 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:20.345726 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:20.345802 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.345807 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.346477 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.346479 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.347081 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.347088 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.347480 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.347482 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.347537 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.347953 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:20.347964 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:20.349131 (+ 1167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:20.350530 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.350533 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.350534 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.350725 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.350995 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.351000 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.351007 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.351080 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.351402 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:20.351403 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:20.351475 (+ 72us) client_negotiation.cc:769] Sending connection context
0925 22:41:20.351561 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 22:41:20.351668 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 22:41:20.351845 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.345243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36864 (local address 127.2.74.252:43589)
0925 22:41:20.345354 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:20.345357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:20.345368 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:41:20.345454 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:20.345456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:20.345491 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:20.345556 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:20.345902 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.346390 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.347172 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.347388 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.347463 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.349252 (+ 1789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:20.349255 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:20.349257 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:20.349277 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:20.350436 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.350807 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.350809 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.350810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.350877 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.351158 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.351160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.351161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.351323 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:20.351397 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:20.351606 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 22:41:20.351721 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":41}
I20260925 22:41:20.355360 9178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b: Bootstrap starting.
I20260925 22:41:20.357792 9178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:20.358597 9178 log.cc:824] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:20.360746 9178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b: No bootstrap required, opened a new log
I20260925 22:41:20.364233 9178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 43589 } }
I20260925 22:41:20.364537 9178 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:20.364632 9178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c135ec404cb42e7b809b5b70986d83b, State: Initialized, Role: FOLLOWER
I20260925 22:41:20.365120 9178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 43589 } }
I20260925 22:41:20.365659 9184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 43589 } } }
I20260925 22:41:20.365782 9184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:41:20.366519 9178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [sys.catalog]: configured and running, proceeding with master startup.
W20260925 22:41:20.372015 9195 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:41:20.372141 9195 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:41:20.375695 9024 master_runner.cc:428] Detected that this master 8c135ec404cb42e7b809b5b70986d83b is joining an existing cluster
I20260925 22:41:20.375787 9024 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.74.252:43589
I20260925 22:41:20.387919 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 8520
I20260925 22:41:20.393839 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:44813
--local_ip_for_outbound_sockets=127.2.74.193
--tserver_master_addrs=127.2.74.254:33523
--webserver_port=42885
--webserver_interface=127.2.74.193
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.193.keytab
--principal=oryx/127.2.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.74.254:33523,127.2.74.253:44125
--tserver_master_addrs=127.2.74.254:33523,127.2.74.253:44125,127.2.74.252:43589 with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:41:20.420465 9203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.379712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:44125 (local address 127.0.0.1:50750)
0925 22:41:20.380233 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:20.380246 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:20.380373 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:20.380794 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:20.380797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:20.381011 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:20.381144 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.381150 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.382306 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.382310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.383425 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.383435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.384234 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.384237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.384327 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.384790 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:20.384805 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:20.416440 (+ 31635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:20.418504 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.418511 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.418515 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.418792 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.419185 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.419188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.419190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.419301 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.419835 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:20.419838 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:20.419942 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 22:41:20.420052 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 22:41:20.420226 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":68,"thread_start_us":97,"threads_started":1}
I20260925 22:41:20.420517 9201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.378476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.0.0.1:52162)
0925 22:41:20.378827 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:20.378845 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:20.378976 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:20.379882 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:20.379886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:20.380222 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:20.380442 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.380448 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.381510 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.381513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.382821 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.382831 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.383611 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.383614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.383822 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.384454 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:20.384477 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:20.415766 (+ 31289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:20.418674 (+ 2908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.418677 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.418680 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.418983 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.419349 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.419353 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.419355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.419451 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.419934 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:20.419937 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:20.420033 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 22:41:20.420128 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 22:41:20.420297 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":311,"spinlock_wait_cycles":14720,"thread_start_us":114,"threads_started":1}
I20260925 22:41:20.421164 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.379366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52162 (local address 127.2.74.254:33523)
0925 22:41:20.379521 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:20.379526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:20.379542 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:41:20.379587 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:20.379590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:20.379665 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:20.379745 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:20.380581 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.381376 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.383011 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.383364 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.383450 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.416169 (+ 32719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:20.416174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:20.416177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:20.416223 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:20.418514 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.419146 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.419150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.419152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.419246 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.419558 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.419561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.419563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.419792 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:20.419889 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:20.420869 (+ 980us) server_negotiation.cc:299] Negotiation successful
0925 22:41:20.421004 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 22:41:20.421208 9182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.379983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50750 (local address 127.2.74.253:44125)
0925 22:41:20.380106 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:20.380111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:20.380312 (+ 201us) server_negotiation.cc:407] Connection header received
0925 22:41:20.380503 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:20.380506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:20.380568 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:20.380622 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:20.381291 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.382146 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.383574 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.383941 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.384020 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.416648 (+ 32628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:20.416653 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:20.416656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:20.416689 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:20.418357 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.418925 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.418929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.418931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.419076 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.419419 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.419422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.419424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.419690 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:20.419790 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:20.420917 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0925 22:41:20.421059 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:41:20.426141 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.379510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36870 (local address 127.2.74.252:43589)
0925 22:41:20.379637 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:20.379642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:20.379657 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:41:20.379700 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:20.379703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:20.379746 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:20.379836 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:20.380564 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.381377 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.382984 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.383343 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.383430 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.421761 (+ 38331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:20.421765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:20.421768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:20.421794 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:20.424497 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.424979 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.424982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.424983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.425055 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.425294 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.425297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.425299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.425490 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:20.425579 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:20.425866 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 22:41:20.425983 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":38}
I20260925 22:41:20.426612 9202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.379038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:43589 (local address 127.0.0.1:36870)
0925 22:41:20.379351 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:20.379363 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:20.379473 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:20.379882 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:20.379886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:20.380222 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:20.380438 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.380453 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.381510 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.381513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.382821 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.382831 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.383611 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.383614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.383866 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.415733 (+ 31867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:20.415755 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:20.417564 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:20.424639 (+ 7075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.424642 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.424644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.424881 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.425144 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.425147 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.425149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.425218 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.425634 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:20.425636 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:20.425715 (+ 79us) client_negotiation.cc:769] Sending connection context
0925 22:41:20.426295 (+ 580us) client_negotiation.cc:240] Negotiation successful
0925 22:41:20.426433 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":273,"spinlock_wait_cycles":16512,"thread_start_us":111,"threads_started":1}
Sep 25 22:41:20 dist-test-slave-csnl krb5kdc[8413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376078, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.254@KRBTEST.COM for oryx/127.2.74.252@KRBTEST.COM
I20260925 22:41:20.435909 9015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.425791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:43589 (local address 127.0.0.1:36872)
0925 22:41:20.426035 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:20.426051 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:20.426129 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:20.426752 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:20.426756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:20.426984 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:20.427091 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.427097 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.428129 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.428133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.428964 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.428975 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.429596 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.429600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.429694 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.430273 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:20.430289 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:20.431988 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:20.434086 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.434090 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.434093 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.434373 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.434730 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.434733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.434735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.434834 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.435320 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:20.435323 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:20.435432 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 22:41:20.435552 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 22:41:20.435724 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260925 22:41:20.437037 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.426277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36872 (local address 127.2.74.252:43589)
0925 22:41:20.426417 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:20.426421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:20.426432 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:41:20.426497 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:20.426500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:20.426562 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:20.426634 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:20.427218 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.427997 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.429114 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.429457 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.429578 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.432177 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:20.432184 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:20.432190 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:20.432225 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:20.433942 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.434490 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.434494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.434496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.434620 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.434937 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.434941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.434942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.435221 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:20.435320 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:20.435619 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 22:41:20.435747 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 22:41:20.439703 8457 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.74.252:43589: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b"
member_type: NON_VOTER
last_known_addr {
host: "127.2.74.252"
port: 43589
}
attrs {
promote: true
}
}
dest_uuid: "f44fda985f2248aa820cd857d80c5e3d"
cas_config_opid_index: 5
I20260925 22:41:20.440371 8457 consensus_queue.cc:237] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: NON_VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: true } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: NON_VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: true } }
I20260925 22:41:20.441507 9020 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8c135ec404cb42e7b809b5b70986d83b (127.2.74.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: NON_VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: true } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: NON_VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: true } } }
I20260925 22:41:20.442719 9152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:20.443138 9020 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.74.252:43589
I20260925 22:41:20.443321 9020 sys_catalog.cc:455] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f44fda985f2248aa820cd857d80c5e3d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: NON_VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: true } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: NON_VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: true } } }
I20260925 22:41:20.443420 9020 sys_catalog.cc:458] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:20.444199 9152 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [term 1 FOLLOWER]: Refusing update from remote peer f44fda985f2248aa820cd857d80c5e3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 22:41:20.444670 9024 master.cc:537] Master@127.2.74.252:43589 shutting down...
I20260925 22:41:20.444864 9021 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: NON_VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 22:41:20.446794 8444 consensus_peers.cc:487] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d -> Peer 8c135ec404cb42e7b809b5b70986d83b (127.2.74.252:43589): Couldn't send request to peer 8c135ec404cb42e7b809b5b70986d83b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 22:41:20.450650 9024 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:41:20.451013 9024 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:41:20.451128 9024 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b: stopping tablet replica
I20260925 22:41:20.453991 9182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.441974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50754 (local address 127.2.74.253:44125)
0925 22:41:20.446361 (+ 4387us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:20.446366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:20.448441 (+ 2075us) server_negotiation.cc:407] Connection header received
0925 22:41:20.448657 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:20.448660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:20.448720 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:20.448785 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:20.449473 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.450326 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.452501 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.453139 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.453226 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.453374 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:20.453765 (+ 391us) server_negotiation.cc:299] Negotiation successful
0925 22:41:20.453849 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4283}
I20260925 22:41:20.454641 9015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.441733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:44125 (local address 127.0.0.1:50754)
0925 22:41:20.448354 (+ 6621us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:20.448372 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:20.448523 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:20.449200 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:20.449206 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:20.449231 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:20.449338 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.449344 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.450730 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.450735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.452330 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.452343 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.453478 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.453482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.453590 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.453640 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 22:41:20.454409 (+ 769us) client_negotiation.cc:240] Negotiation successful
0925 22:41:20.454504 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6524}
I20260925 22:41:20.455502 9070 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [term 1 LEARNER]: Refusing update from remote peer f44fda985f2248aa820cd857d80c5e3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 22:41:20.456019 9021 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: NON_VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 22:41:20.458096 9070 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8c135ec404cb42e7b809b5b70986d83b (127.2.74.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: NON_VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: true } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: NON_VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: true } } }
I20260925 22:41:20.460225 9099 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f44fda985f2248aa820cd857d80c5e3d. Latest consensus state: current_term: 1 leader_uuid: "f44fda985f2248aa820cd857d80c5e3d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: NON_VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: true } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: NON_VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: true } } }
I20260925 22:41:20.460358 9099 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [sys.catalog]: This master's current role is: LEARNER
I20260925 22:41:20.460462 9099 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f44fda985f2248aa820cd857d80c5e3d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: NON_VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: true } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: NON_VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: true } } }
I20260925 22:41:20.460530 9099 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [sys.catalog]: This master's current role is: LEARNER
I20260925 22:41:20.460783 9021 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d: attempting to promote NON_VOTER fbdb039a1ac44df999d6b05de7cb0f77 to VOTER
I20260925 22:41:20.461354 9024 master.cc:552] Master@127.2.74.252:43589 shutdown complete.
I20260925 22:41:20.461259 9021 consensus_queue.cc:237] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: NON_VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: true } }
I20260925 22:41:20.461400 9024 master_runner.cc:305] Clearing existing system tablet
I20260925 22:41:20.461653 9024 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:20.461701 9024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:20.463466 9015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.462657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:43589 (local address 127.0.0.1:36888)
0925 22:41:20.463296 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:20.463379 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:43589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":560}
I20260925 22:41:20.463513 9070 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [term 1 LEARNER]: Refusing update from remote peer f44fda985f2248aa820cd857d80c5e3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 22:41:20.463954 9019 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 22:41:20.464623 9015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.464350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:43589 (local address 127.0.0.1:36900)
0925 22:41:20.464501 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:20.464558 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:43589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260925 22:41:20.464880 8444 consensus_peers.cc:487] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d -> Peer 8c135ec404cb42e7b809b5b70986d83b (127.2.74.252:43589): Couldn't send request to peer 8c135ec404cb42e7b809b5b70986d83b. Status: Network error: Client connection negotiation failed: client connection to 127.2.74.252:43589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 22:41:20.465641 9024 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260925 22:41:20.465901 9019 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, fbdb039a1ac44df999d6b05de7cb0f77 (127.2.74.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: NON_VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: true } } }
I20260925 22:41:20.467108 9208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:20.467342 9024 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:20.467478 9024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 22:41:20.467470 9019 sys_catalog.cc:455] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f44fda985f2248aa820cd857d80c5e3d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: NON_VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: true } } }
I20260925 22:41:20.467512 9024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-2/wal
uuid: "8c135ec404cb42e7b809b5b70986d83b"
format_stamp: "Formatted at 2026-09-25 22:41:20 on dist-test-slave-csnl"
server_key: "1545f0f5184ec273ce477860947598ca"
server_key_iv: "54b928ca903ed184d4b8fab26db1d836"
server_key_version: "encryptionkey@0"
I20260925 22:41:20.467587 9019 sys_catalog.cc:458] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:20.467718 9019 sys_catalog.cc:455] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f44fda985f2248aa820cd857d80c5e3d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: NON_VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: true } } }
I20260925 22:41:20.467788 9019 sys_catalog.cc:458] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:20.468607 9070 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, fbdb039a1ac44df999d6b05de7cb0f77 (127.2.74.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: NON_VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: true } } }
I20260925 22:41:20.469659 9024 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:41:20.470723 9206 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f44fda985f2248aa820cd857d80c5e3d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: NON_VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: true } } }
I20260925 22:41:20.470842 9206 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:41:20.472366 9024 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 22:41:20.472469 9024 log.cc:1196] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 22:41:20.472723 9024 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b: Deleting consensus metadata
I20260925 22:41:20.472877 9024 master_runner.cc:315] Copying system tablet from 127.2.74.254:33523
I20260925 22:41:20.474083 9024 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b: tablet copy: Beginning tablet copy session from remote peer at address 127.2.74.254:33523
I20260925 22:41:20.484952 9214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.474671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.0.0.1:52166)
0925 22:41:20.475012 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:20.475028 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:20.475149 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:20.475490 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:20.475493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:20.475733 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:20.475876 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.475884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.476965 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.476969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.478443 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.478456 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.479368 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.479372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.479476 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.480123 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:20.480139 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:20.480700 (+ 561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:20.483171 (+ 2471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.483175 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.483177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.483479 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.483808 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.483811 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.483813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.483910 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.484358 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:20.484361 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:20.484471 (+ 110us) client_negotiation.cc:769] Sending connection context
0925 22:41:20.484589 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 22:41:20.484752 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":95,"threads_started":1}
I20260925 22:41:20.485345 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.474775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52166 (local address 127.2.74.254:33523)
0925 22:41:20.474920 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:20.474925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:20.475096 (+ 171us) server_negotiation.cc:407] Connection header received
0925 22:41:20.475260 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:20.475263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:20.475334 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:20.475441 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:20.476008 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.476844 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.478614 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.479039 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.479153 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.480898 (+ 1745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:20.480902 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:20.480905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:20.480936 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:20.482870 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.483598 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.483602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.483604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.483704 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.483993 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.483996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.483998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.484258 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:20.484359 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:20.484691 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 22:41:20.484822 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 22:41:20.485690 8477 tablet_copy_service.cc:140] P f44fda985f2248aa820cd857d80c5e3d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8c135ec404cb42e7b809b5b70986d83b ({username='oryx', principal='oryx/127.2.74.252@KRBTEST.COM'} at 127.0.0.1:52166)
I20260925 22:41:20.485795 8477 tablet_copy_service.cc:161] P f44fda985f2248aa820cd857d80c5e3d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8c135ec404cb42e7b809b5b70986d83b at {username='oryx', principal='oryx/127.2.74.252@KRBTEST.COM'} at 127.0.0.1:52166: session id = 8c135ec404cb42e7b809b5b70986d83b-00000000000000000000000000000000
I20260925 22:41:20.486903 8477 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 22:41:20.488442 9024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 22:41:20.491151 9024 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b: tablet copy: Starting download of 0 data blocks...
I20260925 22:41:20.491362 9024 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b: tablet copy: Starting download of 1 WAL segments...
I20260925 22:41:20.493181 9024 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 22:41:20.495441 8477 tablet_copy_service.cc:342] P f44fda985f2248aa820cd857d80c5e3d: Request end of tablet copy session 8c135ec404cb42e7b809b5b70986d83b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.74.252@KRBTEST.COM'} at 127.0.0.1:52166
I20260925 22:41:20.495551 8477 tablet_copy_service.cc:433] P f44fda985f2248aa820cd857d80c5e3d: ending tablet copy session 8c135ec404cb42e7b809b5b70986d83b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8c135ec404cb42e7b809b5b70986d83b
I20260925 22:41:20.504500 9024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:20.506407 9215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:20.507926 9216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:20.508344 9024 server_base.cc:1034] running on GCE node
W20260925 22:41:20.508345 9218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:20.508829 9024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:20.509186 9024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:20.510352 9024 hybrid_clock.cc:648] HybridClock initialized: now 1790376080510323 us; error 46 us; skew 500 ppm
Sep 25 22:41:20 dist-test-slave-csnl krb5kdc[8413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376080, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:20.512156 9024 init.cc:377] Logged in from keytab as oryx/127.2.74.252@KRBTEST.COM (short username oryx)
I20260925 22:41:20.512732 9024 webserver.cc:460] Webserver started at http://127.2.74.252:44271/ using document root <none> and password file <none>
I20260925 22:41:20.512936 9024 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:20.512984 9024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:20.514454 9024 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:41:20.515678 9225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:20.515980 9024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 22:41:20.516065 9024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-2/wal
uuid: "8c135ec404cb42e7b809b5b70986d83b"
format_stamp: "Formatted at 2026-09-25 22:41:20 on dist-test-slave-csnl"
server_key: "1545f0f5184ec273ce477860947598ca"
server_key_iv: "54b928ca903ed184d4b8fab26db1d836"
server_key_version: "encryptionkey@0"
I20260925 22:41:20.516153 9024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:20.533030 9024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:20.535465 9024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:20.540822 9024 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.252:43589
I20260925 22:41:20.540851 9277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.252:43589 every 8 connection(s)
I20260925 22:41:20.541473 9024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/master-2/data/info.pb
I20260925 22:41:20.542939 8783 heartbeater.cc:503] Master 127.2.74.254:33523 was elected leader, sending a full tablet report...
I20260925 22:41:20.543483 9278 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:41:20.544674 9278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b: Bootstrap starting.
W20260925 22:41:20.550664 9204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:20.551069 9204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:20.551148 9204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 22:41:20.556752 9278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:41:20.557071 9278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b: Bootstrap complete.
W20260925 22:41:20.557278 9204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:20.557341 9204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:20.557430 9204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:41:20.557691 9278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: NON_VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: true } }
I20260925 22:41:20.557834 9278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8c135ec404cb42e7b809b5b70986d83b, State: Initialized, Role: LEARNER
I20260925 22:41:20.557928 9278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: NON_VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: true } }
I20260925 22:41:20.558284 9284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: NON_VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: true } } }
I20260925 22:41:20.558398 9278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:41:20.561919 9295 catalog_manager.cc:1259] Loaded cluster ID: 0747fe35ece54cc8b064a2929f65b7e5
I20260925 22:41:20.562057 9295 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b: loading cluster ID for follower catalog manager: success
I20260925 22:41:20.558519 9284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [sys.catalog]: This master's current role is: LEARNER
I20260925 22:41:20.564532 9295 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b: acquiring CA information for follower catalog manager: success
I20260925 22:41:20.565483 9204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.193.keytab
--principal=oryx/127.2.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:44813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=42885
--tserver_master_addrs=127.2.74.254:33523,127.2.74.253:44125,127.2.74.252:43589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:20.566157 9295 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:41:20.566743 9204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:20.567880 9204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:20.578665 9300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:20.578706 9298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:20.578897 9204 server_base.cc:1034] running on GCE node
W20260925 22:41:20.578706 9297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:20.579525 9204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:20.580327 9204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:20.581557 9204 hybrid_clock.cc:648] HybridClock initialized: now 1790376080581522 us; error 35 us; skew 500 ppm
Sep 25 22:41:20 dist-test-slave-csnl krb5kdc[8413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376080, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:20.584643 9204 init.cc:377] Logged in from keytab as oryx/127.2.74.193@KRBTEST.COM (short username oryx)
I20260925 22:41:20.585593 9204 webserver.cc:460] Webserver started at http://127.2.74.193:42885/ using document root <none> and password file <none>
I20260925 22:41:20.586153 9204 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:20.586229 9204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:20.590672 9204 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:41:20.593067 9307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:20.594514 9204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:41:20.594661 9204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "200b7a862e10443ba89a5d209f4959aa"
format_stamp: "Formatted at 2026-09-25 22:41:19 on dist-test-slave-csnl"
server_key: "cb7c709dc942e9c218aa493d9b713e10"
server_key_iv: "738c83a39ffc984b04e8b686866015bc"
server_key_version: "encryptionkey@0"
I20260925 22:41:20.595368 9204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:20.606801 9204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:20.610203 9204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:20.610394 9204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:20.611074 9204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:20.612020 9204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:20.612072 9204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:20.612115 9204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:20.612130 9204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:20.622715 9204 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:44813
I20260925 22:41:20.622803 9420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:44813 every 8 connection(s)
I20260925 22:41:20.623888 9204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:41:20.627944 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 9204
I20260925 22:41:20.628142 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 8654
I20260925 22:41:20.635591 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:38067
--local_ip_for_outbound_sockets=127.2.74.194
--tserver_master_addrs=127.2.74.254:33523
--webserver_port=40505
--webserver_interface=127.2.74.194
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.194.keytab
--principal=oryx/127.2.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.74.254:33523,127.2.74.253:44125
--tserver_master_addrs=127.2.74.254:33523,127.2.74.253:44125,127.2.74.252:43589 with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:41:20 dist-test-slave-csnl krb5kdc[8413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376080, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM
Sep 25 22:41:20 dist-test-slave-csnl krb5kdc[8413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376080, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.252@KRBTEST.COM
Sep 25 22:41:20 dist-test-slave-csnl krb5kdc[8413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376080, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.253@KRBTEST.COM
I20260925 22:41:20.644917 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.627380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:34649 (local address 127.2.74.254:33523)
0925 22:41:20.627509 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:20.627515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:20.627729 (+ 214us) server_negotiation.cc:407] Connection header received
0925 22:41:20.629407 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:20.629414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:20.629479 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:20.629605 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:20.631817 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.632660 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.635159 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.635500 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.635574 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.639491 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:20.639495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:20.639498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:20.639535 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:20.641674 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.642867 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.642873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.642874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.643010 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.643493 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.643497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.643499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.643751 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:20.643836 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:20.644606 (+ 770us) server_negotiation.cc:299] Negotiation successful
0925 22:41:20.644739 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:41:20.645051 9182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.625962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:39493 (local address 127.2.74.253:44125)
0925 22:41:20.626100 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:20.626103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:20.627490 (+ 1387us) server_negotiation.cc:407] Connection header received
0925 22:41:20.629657 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:20.629663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:20.629733 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:20.629816 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:20.631816 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.632731 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.634993 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.635337 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.635474 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.640693 (+ 5219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:20.640702 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:20.640709 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:20.640750 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:20.643167 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.643635 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.643639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.643640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.643719 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.644025 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.644028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.644030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.644236 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:20.644329 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:20.644775 (+ 446us) server_negotiation.cc:299] Negotiation successful
0925 22:41:20.644886 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:41:20.646248 9426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.627261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.2.74.193:34649)
0925 22:41:20.627633 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:20.627646 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:20.629138 (+ 1492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:20.629935 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:20.629945 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:20.630696 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:20.631634 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.631649 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.632849 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.632852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.635041 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.635050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.635873 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.635876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.636014 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.637050 (+ 1036us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:20.637080 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:20.639273 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:20.641829 (+ 2556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.641839 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.641863 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.642733 (+ 870us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.643138 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.643141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.643143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.643361 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.643897 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:20.643905 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:20.644301 (+ 396us) client_negotiation.cc:769] Sending connection context
0925 22:41:20.645546 (+ 1245us) client_negotiation.cc:240] Negotiation successful
0925 22:41:20.645673 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":752,"spinlock_wait_cycles":70144,"thread_start_us":87,"threads_started":1}
I20260925 22:41:20.646552 9427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.627951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:43589 (local address 127.2.74.193:35079)
0925 22:41:20.629620 (+ 1669us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:20.629634 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:20.629759 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:20.630125 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:20.630130 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:20.631392 (+ 1262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:20.631643 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.631649 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.633034 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.633037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.634789 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.634806 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.635679 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.635686 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.636017 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.637050 (+ 1033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:20.637084 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:20.639477 (+ 2393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:20.641862 (+ 2385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.641868 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.641872 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.642725 (+ 853us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.643150 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.643155 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.643159 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.643369 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.643944 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:20.643951 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:20.644303 (+ 352us) client_negotiation.cc:769] Sending connection context
0925 22:41:20.644539 (+ 236us) client_negotiation.cc:240] Negotiation successful
0925 22:41:20.644971 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1604,"mutex_wait_us":394,"spinlock_wait_cycles":20352,"thread_start_us":74,"threads_started":1}
I20260925 22:41:20.647238 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.628072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:35079 (local address 127.2.74.252:43589)
0925 22:41:20.628359 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:20.628366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:20.629696 (+ 1330us) server_negotiation.cc:407] Connection header received
0925 22:41:20.629876 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:20.629879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:20.629930 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:20.629987 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:20.631817 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.632905 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.634994 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.635363 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.635445 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.639645 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:20.639649 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:20.639651 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:20.639681 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:20.641746 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.642870 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.642876 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.642881 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.643014 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.643495 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.643501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.643506 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.643776 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:20.643882 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:20.646780 (+ 2898us) server_negotiation.cc:299] Negotiation successful
0925 22:41:20.646987 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":91,"threads_started":1}
I20260925 22:41:20.646255 9425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.626338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:44125 (local address 127.2.74.193:39493)
0925 22:41:20.627328 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:20.627389 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:20.629377 (+ 1988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:20.629950 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:20.629956 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:20.630929 (+ 973us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:20.631626 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.631650 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.632877 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.632883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.634809 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.634824 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.635694 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.635698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.636146 (+ 448us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.637309 (+ 1163us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:20.637328 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:20.640531 (+ 3203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:20.643283 (+ 2752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.643286 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.643288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.643525 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.643822 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.643825 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.643826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.643918 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.644370 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:20.644373 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:20.644445 (+ 72us) client_negotiation.cc:769] Sending connection context
0925 22:41:20.644539 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 22:41:20.645395 (+ 856us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"mutex_wait_us":1146,"thread_start_us":618,"threads_started":1}
I20260925 22:41:20.648464 9423 heartbeater.cc:347] Connected to a master server at 127.2.74.253:44125
I20260925 22:41:20.648541 9422 heartbeater.cc:347] Connected to a master server at 127.2.74.254:33523
I20260925 22:41:20.648715 9423 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:20.648713 9422 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:20.649106 9421 heartbeater.cc:347] Connected to a master server at 127.2.74.252:43589
I20260925 22:41:20.649163 9421 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:20.649296 9421 heartbeater.cc:511] Master 127.2.74.252:43589 requested a full tablet report, sending...
I20260925 22:41:20.649300 9422 heartbeater.cc:511] Master 127.2.74.254:33523 requested a full tablet report, sending...
I20260925 22:41:20.649366 9423 heartbeater.cc:511] Master 127.2.74.253:44125 requested a full tablet report, sending...
I20260925 22:41:20.650727 8457 ts_manager.cc:194] Re-registered known tserver with Master: 200b7a862e10443ba89a5d209f4959aa (127.2.74.193:44813)
I20260925 22:41:20.651012 9060 ts_manager.cc:194] Registered new tserver with Master: 200b7a862e10443ba89a5d209f4959aa (127.2.74.193:44813)
I20260925 22:41:20.651549 8457 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.74.193@KRBTEST.COM'} at 127.2.74.193:34649
I20260925 22:41:20.651641 9234 ts_manager.cc:194] Registered new tserver with Master: 200b7a862e10443ba89a5d209f4959aa (127.2.74.193:44813)
W20260925 22:41:20.759888 9429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:20.760186 9429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:20.760232 9429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:20.764791 9429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:20.764866 9429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:20.764930 9429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:41:20.767968 8917 heartbeater.cc:503] Master 127.2.74.254:33523 was elected leader, sending a full tablet report...
I20260925 22:41:20.770845 9429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.194.keytab
--principal=oryx/127.2.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:38067
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=40505
--tserver_master_addrs=127.2.74.254:33523,127.2.74.253:44125,127.2.74.252:43589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:20.772083 9429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:20.773178 9429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:41:20.783687 9429 server_base.cc:1034] running on GCE node
W20260925 22:41:20.783619 9434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:20.783625 9435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:20.783622 9437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:20.784472 9429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:20.785261 9429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:20.786450 9429 hybrid_clock.cc:648] HybridClock initialized: now 1790376080786444 us; error 41 us; skew 500 ppm
Sep 25 22:41:20 dist-test-slave-csnl krb5kdc[8413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376080, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:20.789677 9429 init.cc:377] Logged in from keytab as oryx/127.2.74.194@KRBTEST.COM (short username oryx)
I20260925 22:41:20.790745 9429 webserver.cc:460] Webserver started at http://127.2.74.194:40505/ using document root <none> and password file <none>
I20260925 22:41:20.791461 9429 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:20.791577 9429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:20.795820 9429 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:41:20.798031 9444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:20.799523 9429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:41:20.799695 9429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "70289c55941042a9969b16ac4d442f7a"
format_stamp: "Formatted at 2026-09-25 22:41:19 on dist-test-slave-csnl"
server_key: "414e83484c09e3ca1c3f0d8889986895"
server_key_iv: "752ca04abc278b1c554c1ba246b1843b"
server_key_version: "encryptionkey@0"
I20260925 22:41:20.800277 9429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:20.813491 9429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:20.817358 9429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:20.817602 9429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:20.818188 9429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:20.819221 9429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:20.819273 9429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:20.819316 9429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:20.819331 9429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:20.828794 9429 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:38067
I20260925 22:41:20.828843 9557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:38067 every 8 connection(s)
I20260925 22:41:20.829912 9429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:41:20.835290 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 9429
I20260925 22:41:20.835500 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 8788
I20260925 22:41:20.846135 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:37181
--local_ip_for_outbound_sockets=127.2.74.195
--tserver_master_addrs=127.2.74.254:33523
--webserver_port=39241
--webserver_interface=127.2.74.195
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.195.keytab
--principal=oryx/127.2.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.74.254:33523,127.2.74.253:44125
--tserver_master_addrs=127.2.74.254:33523,127.2.74.253:44125,127.2.74.252:43589 with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:41:20 dist-test-slave-csnl krb5kdc[8413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376080, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.252@KRBTEST.COM
Sep 25 22:41:20 dist-test-slave-csnl krb5kdc[8413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376080, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.253@KRBTEST.COM
Sep 25 22:41:20 dist-test-slave-csnl krb5kdc[8413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376080, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM
I20260925 22:41:20.852954 9182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.833734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:49105 (local address 127.2.74.253:44125)
0925 22:41:20.833886 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:20.833894 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:20.834410 (+ 516us) server_negotiation.cc:407] Connection header received
0925 22:41:20.836786 (+ 2376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:20.836790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:20.836855 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:20.836914 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:20.839366 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.840182 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.843066 (+ 2884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.843420 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.843535 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.848455 (+ 4920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:20.848459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:20.848462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:20.848498 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:20.850470 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.851166 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.851169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.851171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.851273 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.851608 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.851611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.851612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.851832 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:20.851907 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:20.852643 (+ 736us) server_negotiation.cc:299] Negotiation successful
0925 22:41:20.852782 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:41:20.852954 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.831907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:57029 (local address 127.2.74.254:33523)
0925 22:41:20.832074 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:20.832079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:20.835530 (+ 3451us) server_negotiation.cc:407] Connection header received
0925 22:41:20.836775 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:20.836779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:20.836869 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:20.836964 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:20.839605 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.840445 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.843751 (+ 3306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.844090 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.844176 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.848296 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:20.848302 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:20.848306 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:20.848340 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:20.850470 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.851166 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.851169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.851171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.851273 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.851605 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.851608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.851610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.851823 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:20.851929 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:20.852642 (+ 713us) server_negotiation.cc:299] Negotiation successful
0925 22:41:20.852782 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 22:41:20.853068 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.833989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:46871 (local address 127.2.74.252:43589)
0925 22:41:20.834169 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:20.834174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:20.838511 (+ 4337us) server_negotiation.cc:407] Connection header received
0925 22:41:20.838635 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:20.838638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:20.838722 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:20.838856 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:20.839825 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.840645 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.842894 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.843397 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.843542 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.847428 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:20.847435 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:20.847440 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:20.847487 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:20.849957 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.850753 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.850758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.850762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.850863 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:20.851405 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:20.851411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:20.851413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:20.851714 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:20.851813 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:20.852667 (+ 854us) server_negotiation.cc:299] Negotiation successful
0925 22:41:20.852841 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 22:41:20.854895 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.833832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:43589 (local address 127.2.74.194:46871)
0925 22:41:20.838396 (+ 4564us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:20.838415 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:20.838539 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:20.839145 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:20.839148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:20.839442 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:20.839616 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.839651 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.840768 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.840771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.842723 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.842736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.843741 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.843748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.843998 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.844899 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:20.844928 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:20.847183 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:20.850125 (+ 2942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.850136 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.850162 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.850609 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.851037 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.851048 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.851052 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.851273 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.851892 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:20.851902 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:20.852322 (+ 420us) client_negotiation.cc:769] Sending connection context
0925 22:41:20.853986 (+ 1664us) client_negotiation.cc:240] Negotiation successful
0925 22:41:20.854179 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4461,"mutex_wait_us":273,"spinlock_wait_cycles":5376,"thread_start_us":91,"threads_started":1}
I20260925 22:41:20.855193 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.832387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:44125 (local address 127.2.74.194:49105)
0925 22:41:20.834207 (+ 1820us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:20.834279 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:20.836439 (+ 2160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:20.837243 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:20.837257 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:20.837950 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:20.839199 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.839222 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.840642 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.840649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.842723 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.842736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.843741 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.843745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.843988 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.845132 (+ 1144us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:20.845149 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:20.848306 (+ 3157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:20.850617 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.850621 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.850623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.851041 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.851381 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.851384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.851386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.851502 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.851968 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:20.851971 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:20.852323 (+ 352us) client_negotiation.cc:769] Sending connection context
0925 22:41:20.853149 (+ 826us) client_negotiation.cc:240] Negotiation successful
0925 22:41:20.853711 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1575,"mutex_wait_us":726,"spinlock_wait_cycles":25600,"thread_start_us":784,"threads_started":1}
I20260925 22:41:20.854892 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.832387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.2.74.194:57029)
0925 22:41:20.835265 (+ 2878us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:20.835314 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:20.836455 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:20.837259 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:20.837263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:20.837970 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:20.839458 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.839467 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.840634 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.840637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.843610 (+ 2973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.843620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.844361 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.844366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.844489 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.845002 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:20.845019 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:20.848102 (+ 3083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:20.850617 (+ 2515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.850621 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.850623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.851041 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.851381 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:20.851384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:20.851386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:20.851497 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:20.851976 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:20.851978 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:20.852323 (+ 345us) client_negotiation.cc:769] Sending connection context
0925 22:41:20.853185 (+ 862us) client_negotiation.cc:240] Negotiation successful
0925 22:41:20.853661 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2720,"mutex_wait_us":567,"spinlock_wait_cycles":59136,"thread_start_us":216,"threads_started":1}
I20260925 22:41:20.856748 9560 heartbeater.cc:347] Connected to a master server at 127.2.74.253:44125
I20260925 22:41:20.856760 9559 heartbeater.cc:347] Connected to a master server at 127.2.74.254:33523
I20260925 22:41:20.856844 9558 heartbeater.cc:347] Connected to a master server at 127.2.74.252:43589
I20260925 22:41:20.857004 9560 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:20.857004 9558 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:20.857004 9559 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:20.857705 9559 heartbeater.cc:511] Master 127.2.74.254:33523 requested a full tablet report, sending...
I20260925 22:41:20.857705 9558 heartbeater.cc:511] Master 127.2.74.252:43589 requested a full tablet report, sending...
I20260925 22:41:20.858395 9560 heartbeater.cc:511] Master 127.2.74.253:44125 requested a full tablet report, sending...
I20260925 22:41:20.859112 8457 ts_manager.cc:194] Re-registered known tserver with Master: 70289c55941042a9969b16ac4d442f7a (127.2.74.194:38067)
I20260925 22:41:20.859272 9060 ts_manager.cc:194] Registered new tserver with Master: 70289c55941042a9969b16ac4d442f7a (127.2.74.194:38067)
I20260925 22:41:20.859643 9234 ts_manager.cc:194] Registered new tserver with Master: 70289c55941042a9969b16ac4d442f7a (127.2.74.194:38067)
I20260925 22:41:20.859855 8457 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.74.194@KRBTEST.COM'} at 127.2.74.194:57029
I20260925 22:41:20.929345 9015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.925393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:43589 (local address 127.0.0.1:36902)
0925 22:41:20.925553 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:20.925572 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:20.925678 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:20.925985 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:20.925987 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:20.926000 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:20.926087 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.926091 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.927184 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.927187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.927952 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:20.927961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.928903 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.928906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:20.928987 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.929008 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:41:20.929063 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 22:41:20.929175 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 22:41:20.929358 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:20.925484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36902 (local address 127.2.74.252:43589)
0925 22:41:20.925641 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:20.925646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:20.925666 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:41:20.925782 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:20.925786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:20.925850 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:20.925964 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:20.926205 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.927053 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.928091 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:20.928765 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:20.928878 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:20.929047 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:20.929084 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 22:41:20.929192 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 22:41:20.930192 9244 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 22:41:20.930668 9284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f44fda985f2248aa820cd857d80c5e3d. Latest consensus state: current_term: 1 leader_uuid: "f44fda985f2248aa820cd857d80c5e3d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: NON_VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: true } } }
I20260925 22:41:20.930850 9284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [sys.catalog]: This master's current role is: LEARNER
W20260925 22:41:20.969323 9565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:20.969630 9565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:20.969676 9565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:20.973992 9565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:20.974071 9565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:20.974135 9565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:41:20.979854 9565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.195.keytab
--principal=oryx/127.2.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:37181
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=39241
--tserver_master_addrs=127.2.74.254:33523,127.2.74.253:44125,127.2.74.252:43589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:20.981005 9565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:20.982070 9565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:20.992303 9573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:20.992319 9571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:20.992307 9570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:20.992527 9565 server_base.cc:1034] running on GCE node
I20260925 22:41:20.993060 9565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:20.993777 9565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:20.995003 9565 hybrid_clock.cc:648] HybridClock initialized: now 1790376080994979 us; error 38 us; skew 500 ppm
Sep 25 22:41:20 dist-test-slave-csnl krb5kdc[8413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376080, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:20.998070 9565 init.cc:377] Logged in from keytab as oryx/127.2.74.195@KRBTEST.COM (short username oryx)
I20260925 22:41:20.999217 9565 webserver.cc:460] Webserver started at http://127.2.74.195:39241/ using document root <none> and password file <none>
I20260925 22:41:20.999900 9565 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:21.000025 9565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:21.004508 9565 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:41:21.007247 9580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:21.008863 9565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:41:21.009060 9565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "e74fb20181524e049aca671bdc2d4bbd"
format_stamp: "Formatted at 2026-09-25 22:41:19 on dist-test-slave-csnl"
server_key: "ca5b5d64dfa2a0835e7cd825641ef53f"
server_key_iv: "d523020d106d3670f3e0c7674e9a92aa"
server_key_version: "encryptionkey@0"
I20260925 22:41:21.009752 9565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:21.025635 9565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:21.029239 9565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:21.029467 9565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:21.030122 9565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:21.031136 9565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:21.031204 9565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:21.031278 9565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:21.031319 9565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:21.041617 9565 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:37181
I20260925 22:41:21.041633 9693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:37181 every 8 connection(s)
I20260925 22:41:21.042683 9565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:41:21.044179 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 9565
Sep 25 22:41:21 dist-test-slave-csnl krb5kdc[8413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376080, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.253@KRBTEST.COM
Sep 25 22:41:21 dist-test-slave-csnl krb5kdc[8413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376080, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.252@KRBTEST.COM
I20260925 22:41:21.060261 9705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:21.046628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50756 (local address 127.2.74.253:44125)
0925 22:41:21.047272 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:21.047277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:21.047762 (+ 485us) server_negotiation.cc:407] Connection header received
0925 22:41:21.047976 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:21.047979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:21.048022 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:21.048109 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:21.049129 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.049930 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.051164 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.051521 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.051606 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:21.054089 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:21.054096 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:21.054100 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:21.054126 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:21.056162 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:21.057901 (+ 1739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:21.057909 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:21.057914 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:21.058029 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:21.058518 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:21.058523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:21.058527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:21.058775 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:21.058907 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:21.059827 (+ 920us) server_negotiation.cc:299] Negotiation successful
0925 22:41:21.059996 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":84,"threads_started":1}
Sep 25 22:41:21 dist-test-slave-csnl krb5kdc[8413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376080, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM
I20260925 22:41:21.063545 9708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:21.047263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52182 (local address 127.2.74.254:33523)
0925 22:41:21.047501 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:21.047512 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:21.048507 (+ 995us) server_negotiation.cc:407] Connection header received
0925 22:41:21.049363 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:21.049366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:21.049423 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:21.049506 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:21.052625 (+ 3119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.053487 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.056476 (+ 2989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.056834 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.056945 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:21.058843 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:21.058848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:21.058851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:21.058884 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:21.061072 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:21.061616 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:21.061623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:21.061625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:21.061718 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:21.062116 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:21.062119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:21.062121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:21.062358 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:21.062501 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:21.063172 (+ 671us) server_negotiation.cc:299] Negotiation successful
0925 22:41:21.063308 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":83,"threads_started":1}
I20260925 22:41:21.066828 9182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:21.044810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:32891 (local address 127.2.74.253:44125)
0925 22:41:21.044979 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:21.044984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:21.046065 (+ 1081us) server_negotiation.cc:407] Connection header received
0925 22:41:21.047479 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:21.047483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:21.047540 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:21.047609 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:21.049628 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.050334 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.052206 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.052562 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.052645 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:21.057102 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:21.057106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:21.057109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:21.057133 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:21.063785 (+ 6652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:21.064600 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:21.064604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:21.064606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:21.064856 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:21.065387 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:21.065391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:21.065393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:21.065628 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:21.065707 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:21.066500 (+ 793us) server_negotiation.cc:299] Negotiation successful
0925 22:41:21.066634 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:41:21.067225 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:21.047059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:49867 (local address 127.2.74.254:33523)
0925 22:41:21.047199 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:21.047206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:21.056823 (+ 9617us) server_negotiation.cc:407] Connection header received
0925 22:41:21.057030 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:21.057034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:21.057085 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:21.057148 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:21.057876 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.058771 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.060109 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.060455 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.060555 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:21.063239 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:21.063243 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:21.063245 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:21.063268 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:21.065044 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:21.065525 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:21.065541 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:21.065542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:21.065619 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:21.066169 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:21.066172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:21.066174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:21.066378 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:21.066470 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:21.066797 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 22:41:21.067091 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:41:21.068110 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:21.047514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36904 (local address 127.2.74.252:43589)
0925 22:41:21.047771 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:21.047783 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:21.049031 (+ 1248us) server_negotiation.cc:407] Connection header received
0925 22:41:21.049074 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:21.049077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:21.049122 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:21.049213 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:21.051607 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.052451 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.055891 (+ 3440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.056166 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.056300 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:21.062128 (+ 5828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:21.062132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:21.062135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:21.062168 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:21.064260 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:21.064837 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:21.064844 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:21.064847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:21.064949 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:21.065286 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:21.065291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:21.065292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:21.065542 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:21.065676 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:21.066404 (+ 728us) server_negotiation.cc:299] Negotiation successful
0925 22:41:21.067924 (+ 1520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":99,"threads_started":1}
I20260925 22:41:21.068604 9706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:21.046937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.2.74.195:49867)
0925 22:41:21.056721 (+ 9784us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:21.056742 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:21.056905 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:21.057318 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:21.057324 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:21.057571 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:21.057743 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:21.057755 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.058930 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.058933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:21.059916 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:21.059928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.060771 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.060777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:21.060891 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:21.061452 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:21.061467 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:21.063087 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:21.065173 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:21.065175 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:21.065179 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:21.065420 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:21.065732 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:21.065736 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:21.065740 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:21.065821 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:21.066524 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:21.066539 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:21.066623 (+ 84us) client_negotiation.cc:769] Sending connection context
0925 22:41:21.067413 (+ 790us) client_negotiation.cc:240] Negotiation successful
0925 22:41:21.067540 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9668,"thread_start_us":85,"threads_started":1}
I20260925 22:41:21.068604 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:21.045323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:44125 (local address 127.2.74.195:32891)
0925 22:41:21.045884 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:21.045942 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:21.047202 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:21.048123 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:21.048132 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:21.048674 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:21.049468 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:21.049487 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.050499 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.050502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:21.052075 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:21.052086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.052849 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.052853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:21.053163 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:21.053981 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:21.054012 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:21.056893 (+ 2881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:21.063953 (+ 7060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:21.063961 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:21.063988 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:21.064461 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:21.064993 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:21.064998 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:21.065003 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:21.065265 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:21.065770 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:21.065781 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:21.066191 (+ 410us) client_negotiation.cc:769] Sending connection context
0925 22:41:21.067054 (+ 863us) client_negotiation.cc:240] Negotiation successful
0925 22:41:21.067451 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":129,"spinlock_wait_cycles":4992,"thread_start_us":173,"threads_started":1}
I20260925 22:41:21.070097 9695 heartbeater.cc:347] Connected to a master server at 127.2.74.254:33523
I20260925 22:41:21.070313 9696 heartbeater.cc:347] Connected to a master server at 127.2.74.253:44125
I20260925 22:41:21.070374 9696 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:21.070374 9695 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:21.071040 9696 heartbeater.cc:511] Master 127.2.74.253:44125 requested a full tablet report, sending...
I20260925 22:41:21.071040 9695 heartbeater.cc:511] Master 127.2.74.254:33523 requested a full tablet report, sending...
I20260925 22:41:21.071201 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:21.046411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:50783 (local address 127.2.74.252:43589)
0925 22:41:21.046574 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:21.046579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:21.048232 (+ 1653us) server_negotiation.cc:407] Connection header received
0925 22:41:21.048456 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:21.048459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:21.048524 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:21.048589 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:21.049635 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.050541 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.052213 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.052617 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.052691 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:21.058710 (+ 6019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:21.058715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:21.058717 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:21.058745 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:21.066680 (+ 7935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:21.067608 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:21.067612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:21.067614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:21.067705 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:21.070071 (+ 2366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:21.070075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:21.070077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:21.070303 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:21.070391 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:21.070866 (+ 475us) server_negotiation.cc:299] Negotiation successful
0925 22:41:21.071037 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 22:41:21.071936 9703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:21.045324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:43589 (local address 127.2.74.195:50783)
0925 22:41:21.048129 (+ 2805us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:21.048144 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:21.048325 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:21.048750 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:21.048756 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:21.049038 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:21.049472 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:21.049492 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.050708 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.050712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:21.052069 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:21.052081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.052853 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.052857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:21.053163 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:21.054049 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:21.054065 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:21.057195 (+ 3130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:21.066811 (+ 9616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:21.066817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:21.066823 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:21.067478 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:21.069810 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:21.069814 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:21.069816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:21.069934 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:21.070566 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:21.070569 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:21.070674 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 22:41:21.071573 (+ 899us) client_negotiation.cc:240] Negotiation successful
0925 22:41:21.071741 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2708,"mutex_wait_us":154,"spinlock_wait_cycles":17280,"thread_start_us":153,"threads_started":1}
I20260925 22:41:21.068478 8457 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 22:41:21.072183 8458 ts_manager.cc:194] Re-registered known tserver with Master: e74fb20181524e049aca671bdc2d4bbd (127.2.74.195:37181)
I20260925 22:41:21.072304 9060 ts_manager.cc:194] Registered new tserver with Master: e74fb20181524e049aca671bdc2d4bbd (127.2.74.195:37181)
I20260925 22:41:21.072932 8458 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.74.195@KRBTEST.COM'} at 127.2.74.195:49867
I20260925 22:41:21.073088 9694 heartbeater.cc:347] Connected to a master server at 127.2.74.252:43589
I20260925 22:41:21.073161 9694 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:21.073340 9694 heartbeater.cc:511] Master 127.2.74.252:43589 requested a full tablet report, sending...
W20260925 22:41:21.073348 8457 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:41:21.074137 9234 ts_manager.cc:194] Registered new tserver with Master: e74fb20181524e049aca671bdc2d4bbd (127.2.74.195:37181)
I20260925 22:41:21.076256 9021 sys_catalog.cc:455] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f44fda985f2248aa820cd857d80c5e3d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: NON_VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: true } } }
I20260925 22:41:21.076427 9021 sys_catalog.cc:458] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:21.080297 9021 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d: attempting to promote NON_VOTER 8c135ec404cb42e7b809b5b70986d83b to VOTER
I20260925 22:41:21.080726 9021 consensus_queue.cc:237] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: false } }
I20260925 22:41:21.082096 9070 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [term 1 FOLLOWER]: Refusing update from remote peer f44fda985f2248aa820cd857d80c5e3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 22:41:21.082484 9714 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 22:41:21.083918 9244 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [term 1 LEARNER]: Refusing update from remote peer f44fda985f2248aa820cd857d80c5e3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 22:41:21.084679 9712 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 22:41:21.085094 9714 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8c135ec404cb42e7b809b5b70986d83b (127.2.74.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: false } } }
I20260925 22:41:21.086144 9244 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8c135ec404cb42e7b809b5b70986d83b (127.2.74.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: false } } }
I20260925 22:41:21.086561 9714 sys_catalog.cc:455] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f44fda985f2248aa820cd857d80c5e3d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: false } } }
I20260925 22:41:21.086675 9714 sys_catalog.cc:458] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:21.086648 9712 sys_catalog.cc:455] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f44fda985f2248aa820cd857d80c5e3d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: false } } }
I20260925 22:41:21.086797 9712 sys_catalog.cc:458] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:21.087311 9070 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8c135ec404cb42e7b809b5b70986d83b (127.2.74.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: false } } }
I20260925 22:41:21.087950 9284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f44fda985f2248aa820cd857d80c5e3d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: false } } }
I20260925 22:41:21.088052 9284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:41:21.088569 9713 sys_catalog.cc:455] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f44fda985f2248aa820cd857d80c5e3d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: false } } }
I20260925 22:41:21.088683 9713 sys_catalog.cc:458] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:21.093312 9722 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f44fda985f2248aa820cd857d80c5e3d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: false } } }
I20260925 22:41:21.093470 9722 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:41:21.096185 9015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:21.091010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:44813 (local address 127.0.0.1:46792)
0925 22:41:21.091136 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:21.091152 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:21.091235 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:21.092168 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:21.092172 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:21.092193 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:21.092308 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:21.092314 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.093798 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.093803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:21.094618 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:21.094626 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.095793 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.095797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:21.095896 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:21.095924 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 22:41:21.095974 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 22:41:21.096043 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 22:41:21.096973 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:21.091198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46792 (local address 127.2.74.193:44813)
0925 22:41:21.091644 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:21.091658 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:21.091682 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:41:21.091763 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:21.091770 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:21.092021 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:21.092205 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:21.092472 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.093668 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.094749 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.095647 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.096499 (+ 852us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:21.096666 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:21.096722 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 22:41:21.096822 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":111,"threads_started":1}
I20260925 22:41:21.097021 9724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:21.091290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:37181 (local address 127.0.0.1:54778)
0925 22:41:21.092150 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:21.092164 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:21.092238 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:21.092925 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:21.092928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:21.092945 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:21.093038 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:21.093047 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.094543 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.094547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:21.095640 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:21.095647 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.096623 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.096627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:21.096721 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:21.096743 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:41:21.096821 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 22:41:21.096889 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":121,"threads_started":1}
I20260925 22:41:21.097056 9725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:21.091512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54778 (local address 127.2.74.195:37181)
0925 22:41:21.092497 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:21.092511 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:21.092537 (+ 26us) server_negotiation.cc:407] Connection header received
0925 22:41:21.092610 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:21.092617 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:21.092789 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:21.092935 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:21.093162 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.094406 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.095779 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.096492 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.096627 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:21.096768 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:21.096849 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 22:41:21.096932 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":885,"thread_start_us":92,"threads_started":1}
I20260925 22:41:21.097856 9726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:21.091682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:38067 (local address 127.0.0.1:44088)
0925 22:41:21.092996 (+ 1314us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:21.093011 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:21.093108 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:21.093536 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:21.093539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:21.093560 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:21.093642 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:21.093648 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.095146 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.095150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:21.096352 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:21.096364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.097484 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.097488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:21.097579 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:21.097603 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:41:21.097654 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 22:41:21.097721 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1237,"thread_start_us":102,"threads_started":1}
I20260925 22:41:21.098481 9727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:21.091843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44088 (local address 127.2.74.194:38067)
0925 22:41:21.092609 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:21.092620 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:21.093088 (+ 468us) server_negotiation.cc:407] Connection header received
0925 22:41:21.093215 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:21.093220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:21.093386 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:21.093775 (+ 389us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:21.093913 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.095025 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.096499 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.097339 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.098138 (+ 799us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:21.098250 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:21.098292 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 22:41:21.098362 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":107,"threads_started":1}
I20260925 22:41:21.100670 9355 tablet_service.cc:1467] Processing CreateTablet for tablet 463dee26894f4e99a190080b870962db (DEFAULT_TABLE table=test-table [id=210b048a2bbe4a7ea26861b2a2cf763a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:21.100955 9628 tablet_service.cc:1467] Processing CreateTablet for tablet 463dee26894f4e99a190080b870962db (DEFAULT_TABLE table=test-table [id=210b048a2bbe4a7ea26861b2a2cf763a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:21.100980 9492 tablet_service.cc:1467] Processing CreateTablet for tablet 463dee26894f4e99a190080b870962db (DEFAULT_TABLE table=test-table [id=210b048a2bbe4a7ea26861b2a2cf763a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:21.101825 9492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 463dee26894f4e99a190080b870962db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:21.101831 9355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 463dee26894f4e99a190080b870962db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:21.101831 9628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 463dee26894f4e99a190080b870962db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:21.109649 9728 tablet_bootstrap.cc:492] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a: Bootstrap starting.
I20260925 22:41:21.110481 9730 tablet_bootstrap.cc:492] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd: Bootstrap starting.
I20260925 22:41:21.110678 9729 tablet_bootstrap.cc:492] T 463dee26894f4e99a190080b870962db P 200b7a862e10443ba89a5d209f4959aa: Bootstrap starting.
I20260925 22:41:21.112396 9730 tablet_bootstrap.cc:654] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:21.112613 9728 tablet_bootstrap.cc:654] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:21.112808 9729 tablet_bootstrap.cc:654] T 463dee26894f4e99a190080b870962db P 200b7a862e10443ba89a5d209f4959aa: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:21.113245 9730 log.cc:824] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:21.113518 9728 log.cc:824] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:21.113548 9729 log.cc:824] T 463dee26894f4e99a190080b870962db P 200b7a862e10443ba89a5d209f4959aa: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:21.116628 9729 tablet_bootstrap.cc:492] T 463dee26894f4e99a190080b870962db P 200b7a862e10443ba89a5d209f4959aa: No bootstrap required, opened a new log
I20260925 22:41:21.116873 9730 tablet_bootstrap.cc:492] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd: No bootstrap required, opened a new log
I20260925 22:41:21.116868 9729 ts_tablet_manager.cc:1397] T 463dee26894f4e99a190080b870962db P 200b7a862e10443ba89a5d209f4959aa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:41:21.117076 9730 ts_tablet_manager.cc:1397] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 22:41:21.120434 9728 tablet_bootstrap.cc:492] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a: No bootstrap required, opened a new log
I20260925 22:41:21.120693 9728 ts_tablet_manager.cc:1397] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.005s
I20260925 22:41:21.122668 9729 raft_consensus.cc:348] T 463dee26894f4e99a190080b870962db P 200b7a862e10443ba89a5d209f4959aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200b7a862e10443ba89a5d209f4959aa" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44813 } } peers { permanent_uuid: "e74fb20181524e049aca671bdc2d4bbd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 37181 } } peers { permanent_uuid: "70289c55941042a9969b16ac4d442f7a" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38067 } }
I20260925 22:41:21.122668 9730 raft_consensus.cc:348] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200b7a862e10443ba89a5d209f4959aa" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44813 } } peers { permanent_uuid: "e74fb20181524e049aca671bdc2d4bbd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 37181 } } peers { permanent_uuid: "70289c55941042a9969b16ac4d442f7a" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38067 } }
I20260925 22:41:21.123042 9730 raft_consensus.cc:374] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:21.123042 9729 raft_consensus.cc:374] T 463dee26894f4e99a190080b870962db P 200b7a862e10443ba89a5d209f4959aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:21.123099 9729 raft_consensus.cc:729] T 463dee26894f4e99a190080b870962db P 200b7a862e10443ba89a5d209f4959aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 200b7a862e10443ba89a5d209f4959aa, State: Initialized, Role: FOLLOWER
I20260925 22:41:21.123100 9730 raft_consensus.cc:729] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e74fb20181524e049aca671bdc2d4bbd, State: Initialized, Role: FOLLOWER
I20260925 22:41:21.123652 9729 consensus_queue.cc:260] T 463dee26894f4e99a190080b870962db P 200b7a862e10443ba89a5d209f4959aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200b7a862e10443ba89a5d209f4959aa" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44813 } } peers { permanent_uuid: "e74fb20181524e049aca671bdc2d4bbd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 37181 } } peers { permanent_uuid: "70289c55941042a9969b16ac4d442f7a" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38067 } }
I20260925 22:41:21.123653 9730 consensus_queue.cc:260] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200b7a862e10443ba89a5d209f4959aa" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44813 } } peers { permanent_uuid: "e74fb20181524e049aca671bdc2d4bbd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 37181 } } peers { permanent_uuid: "70289c55941042a9969b16ac4d442f7a" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38067 } }
I20260925 22:41:21.124884 9730 ts_tablet_manager.cc:1428] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260925 22:41:21.124884 9729 ts_tablet_manager.cc:1428] T 463dee26894f4e99a190080b870962db P 200b7a862e10443ba89a5d209f4959aa: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260925 22:41:21.125116 9422 heartbeater.cc:503] Master 127.2.74.254:33523 was elected leader, sending a full tablet report...
I20260925 22:41:21.125453 9728 raft_consensus.cc:348] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200b7a862e10443ba89a5d209f4959aa" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44813 } } peers { permanent_uuid: "e74fb20181524e049aca671bdc2d4bbd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 37181 } } peers { permanent_uuid: "70289c55941042a9969b16ac4d442f7a" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38067 } }
I20260925 22:41:21.125756 9728 raft_consensus.cc:374] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:21.125816 9728 raft_consensus.cc:729] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70289c55941042a9969b16ac4d442f7a, State: Initialized, Role: FOLLOWER
I20260925 22:41:21.126338 9728 consensus_queue.cc:260] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200b7a862e10443ba89a5d209f4959aa" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44813 } } peers { permanent_uuid: "e74fb20181524e049aca671bdc2d4bbd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 37181 } } peers { permanent_uuid: "70289c55941042a9969b16ac4d442f7a" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38067 } }
I20260925 22:41:21.127302 9695 heartbeater.cc:503] Master 127.2.74.254:33523 was elected leader, sending a full tablet report...
W20260925 22:41:21.127431 9424 tablet.cc:2369] T 463dee26894f4e99a190080b870962db P 200b7a862e10443ba89a5d209f4959aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:41:21.130560 9559 heartbeater.cc:503] Master 127.2.74.254:33523 was elected leader, sending a full tablet report...
I20260925 22:41:21.132710 9728 ts_tablet_manager.cc:1428] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
W20260925 22:41:21.298511 9697 tablet.cc:2369] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:41:21.304066 9736 raft_consensus.cc:482] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:41:21.304315 9736 raft_consensus.cc:504] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200b7a862e10443ba89a5d209f4959aa" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44813 } } peers { permanent_uuid: "e74fb20181524e049aca671bdc2d4bbd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 37181 } } peers { permanent_uuid: "70289c55941042a9969b16ac4d442f7a" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38067 } }
I20260925 22:41:21.305115 9734 raft_consensus.cc:482] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:41:21.305297 9734 raft_consensus.cc:504] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200b7a862e10443ba89a5d209f4959aa" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44813 } } peers { permanent_uuid: "e74fb20181524e049aca671bdc2d4bbd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 37181 } } peers { permanent_uuid: "70289c55941042a9969b16ac4d442f7a" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38067 } }
I20260925 22:41:21.305789 9736 leader_election.cc:288] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 200b7a862e10443ba89a5d209f4959aa (127.2.74.193:44813), e74fb20181524e049aca671bdc2d4bbd (127.2.74.195:37181)
I20260925 22:41:21.309865 9734 leader_election.cc:288] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 200b7a862e10443ba89a5d209f4959aa (127.2.74.193:44813), 70289c55941042a9969b16ac4d442f7a (127.2.74.194:38067)
I20260925 22:41:21.310563 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:21.306114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:51731 (local address 127.2.74.193:44813)
0925 22:41:21.306263 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:21.306270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:21.306294 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:41:21.306491 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:21.306495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:21.306553 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:21.306625 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:21.307063 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.307802 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.309336 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.309941 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.310060 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:21.310113 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:21.310349 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 22:41:21.310430 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:41:21.310935 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:21.306055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:44813 (local address 127.2.74.194:51731)
0925 22:41:21.306212 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:21.306229 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:21.306343 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:21.306752 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:21.306755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:21.306773 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:21.306908 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:21.306916 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.307929 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.307932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:21.309198 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:21.309209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.310069 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.310072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:21.310188 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:21.310212 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:41:21.310280 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 22:41:21.310752 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":384}
I20260925 22:41:21.311327 9703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:21.306706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:44813 (local address 127.2.74.195:37435)
0925 22:41:21.306876 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:21.306891 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:21.307022 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:21.307672 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:21.307675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:21.307703 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:21.307847 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:21.307857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.308940 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.308943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:21.310068 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:21.310076 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.310907 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.310910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:21.311047 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:21.311079 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 22:41:21.311136 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 22:41:21.311224 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260925 22:41:21.311570 9706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:21.306744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:38067 (local address 127.2.74.195:56297)
0925 22:41:21.306939 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:21.306975 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:21.307025 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:21.307331 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:21.307333 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:21.307367 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:21.307487 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:21.307493 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.308464 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.308467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:21.309688 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:21.309698 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.311162 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.311166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:21.311268 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:21.311290 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:41:21.311371 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 22:41:21.311446 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":8,"spinlock_wait_cycles":9856}
I20260925 22:41:21.311605 9727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:21.306983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:56297 (local address 127.2.74.194:38067)
0925 22:41:21.307087 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:21.307091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:21.307102 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:41:21.307134 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:21.307136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:21.307174 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:21.307239 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:21.307615 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.308319 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.309822 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.310874 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.310994 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:21.311041 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:21.311422 (+ 381us) server_negotiation.cc:299] Negotiation successful
0925 22:41:21.311500 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":446,"server-negotiator.queue_time_us":43}
I20260925 22:41:21.311657 9375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "463dee26894f4e99a190080b870962db" candidate_uuid: "70289c55941042a9969b16ac4d442f7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "200b7a862e10443ba89a5d209f4959aa" is_pre_election: true
I20260925 22:41:21.311836 9737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:21.306737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:37435 (local address 127.2.74.193:44813)
0925 22:41:21.307340 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:21.307344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:21.307355 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:41:21.307405 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:21.307411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:21.307554 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:21.307972 (+ 418us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:21.308099 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.308829 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.310198 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.310761 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.311540 (+ 779us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:21.311583 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:21.311619 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 22:41:21.311693 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":93,"threads_started":1}
I20260925 22:41:21.312031 9375 raft_consensus.cc:2454] T 463dee26894f4e99a190080b870962db P 200b7a862e10443ba89a5d209f4959aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70289c55941042a9969b16ac4d442f7a in term 0.
I20260925 22:41:21.312355 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "463dee26894f4e99a190080b870962db" candidate_uuid: "e74fb20181524e049aca671bdc2d4bbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "200b7a862e10443ba89a5d209f4959aa" is_pre_election: true
I20260925 22:41:21.312497 9374 raft_consensus.cc:2454] T 463dee26894f4e99a190080b870962db P 200b7a862e10443ba89a5d209f4959aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e74fb20181524e049aca671bdc2d4bbd in term 0.
I20260925 22:41:21.312704 9512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "463dee26894f4e99a190080b870962db" candidate_uuid: "e74fb20181524e049aca671bdc2d4bbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70289c55941042a9969b16ac4d442f7a" is_pre_election: true
I20260925 22:41:21.312904 9512 raft_consensus.cc:2454] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e74fb20181524e049aca671bdc2d4bbd in term 0.
I20260925 22:41:21.312887 9584 leader_election.cc:302] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 200b7a862e10443ba89a5d209f4959aa, e74fb20181524e049aca671bdc2d4bbd; no voters:
I20260925 22:41:21.312883 9448 leader_election.cc:302] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 200b7a862e10443ba89a5d209f4959aa, 70289c55941042a9969b16ac4d442f7a; no voters:
I20260925 22:41:21.312923 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:21.306341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:37181 (local address 127.2.74.194:35031)
0925 22:41:21.306432 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:21.306446 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:21.306525 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:21.308478 (+ 1953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:21.308482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:21.308500 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:21.308607 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:21.308612 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.309643 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.309647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:21.310814 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:21.310827 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.312518 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.312521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:21.312639 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:21.312659 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 22:41:21.312724 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 22:41:21.312795 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":13}
I20260925 22:41:21.313149 9725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:21.307153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:35031 (local address 127.2.74.195:37181)
0925 22:41:21.307268 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:21.307272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:21.307285 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:41:21.307324 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:21.307327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:21.307385 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:21.307455 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:21.308782 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.309509 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.311793 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.312412 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.312516 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:21.312559 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:21.312970 (+ 411us) server_negotiation.cc:299] Negotiation successful
0925 22:41:21.313042 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":46,"spinlock_wait_cycles":5888}
I20260925 22:41:21.313328 9736 raft_consensus.cc:2781] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:41:21.313434 9736 raft_consensus.cc:482] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:41:21.313485 9736 raft_consensus.cc:3037] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:21.313580 9734 raft_consensus.cc:2781] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:41:21.313573 9648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "463dee26894f4e99a190080b870962db" candidate_uuid: "70289c55941042a9969b16ac4d442f7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e74fb20181524e049aca671bdc2d4bbd" is_pre_election: true
I20260925 22:41:21.313886 9734 raft_consensus.cc:482] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:41:21.313946 9734 raft_consensus.cc:3037] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:21.314775 9736 raft_consensus.cc:504] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200b7a862e10443ba89a5d209f4959aa" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44813 } } peers { permanent_uuid: "e74fb20181524e049aca671bdc2d4bbd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 37181 } } peers { permanent_uuid: "70289c55941042a9969b16ac4d442f7a" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38067 } }
I20260925 22:41:21.315083 9734 raft_consensus.cc:504] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200b7a862e10443ba89a5d209f4959aa" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44813 } } peers { permanent_uuid: "e74fb20181524e049aca671bdc2d4bbd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 37181 } } peers { permanent_uuid: "70289c55941042a9969b16ac4d442f7a" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38067 } }
I20260925 22:41:21.315196 9736 leader_election.cc:288] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a [CANDIDATE]: Term 1 election: Requested vote from peers 200b7a862e10443ba89a5d209f4959aa (127.2.74.193:44813), e74fb20181524e049aca671bdc2d4bbd (127.2.74.195:37181)
I20260925 22:41:21.315549 9374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "463dee26894f4e99a190080b870962db" candidate_uuid: "70289c55941042a9969b16ac4d442f7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "200b7a862e10443ba89a5d209f4959aa"
I20260925 22:41:21.315667 9374 raft_consensus.cc:3037] T 463dee26894f4e99a190080b870962db P 200b7a862e10443ba89a5d209f4959aa [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:21.315336 9648 raft_consensus.cc:2379] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 70289c55941042a9969b16ac4d442f7a in current term 1: Already voted for candidate e74fb20181524e049aca671bdc2d4bbd in this term.
I20260925 22:41:21.315971 9734 leader_election.cc:288] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd [CANDIDATE]: Term 1 election: Requested vote from peers 200b7a862e10443ba89a5d209f4959aa (127.2.74.193:44813), 70289c55941042a9969b16ac4d442f7a (127.2.74.194:38067)
I20260925 22:41:21.316825 9374 raft_consensus.cc:2454] T 463dee26894f4e99a190080b870962db P 200b7a862e10443ba89a5d209f4959aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70289c55941042a9969b16ac4d442f7a in term 1.
I20260925 22:41:21.316859 9648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "463dee26894f4e99a190080b870962db" candidate_uuid: "70289c55941042a9969b16ac4d442f7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e74fb20181524e049aca671bdc2d4bbd"
I20260925 22:41:21.316862 9512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "463dee26894f4e99a190080b870962db" candidate_uuid: "e74fb20181524e049aca671bdc2d4bbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70289c55941042a9969b16ac4d442f7a"
I20260925 22:41:21.316980 9648 raft_consensus.cc:2379] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 70289c55941042a9969b16ac4d442f7a in current term 1: Already voted for candidate e74fb20181524e049aca671bdc2d4bbd in this term.
I20260925 22:41:21.317008 9512 raft_consensus.cc:2379] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e74fb20181524e049aca671bdc2d4bbd in current term 1: Already voted for candidate 70289c55941042a9969b16ac4d442f7a in this term.
I20260925 22:41:21.317116 9448 leader_election.cc:302] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 200b7a862e10443ba89a5d209f4959aa, 70289c55941042a9969b16ac4d442f7a; no voters:
I20260925 22:41:21.317200 9375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "463dee26894f4e99a190080b870962db" candidate_uuid: "e74fb20181524e049aca671bdc2d4bbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "200b7a862e10443ba89a5d209f4959aa"
I20260925 22:41:21.317291 9736 raft_consensus.cc:2781] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:41:21.317353 9375 raft_consensus.cc:2379] T 463dee26894f4e99a190080b870962db P 200b7a862e10443ba89a5d209f4959aa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e74fb20181524e049aca671bdc2d4bbd in current term 1: Already voted for candidate 70289c55941042a9969b16ac4d442f7a in this term.
I20260925 22:41:21.317508 9736 raft_consensus.cc:686] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a [term 1 LEADER]: Becoming Leader. State: Replica: 70289c55941042a9969b16ac4d442f7a, State: Running, Role: LEADER
I20260925 22:41:21.317888 9584 leader_election.cc:302] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e74fb20181524e049aca671bdc2d4bbd; no voters: 200b7a862e10443ba89a5d209f4959aa, 70289c55941042a9969b16ac4d442f7a
I20260925 22:41:21.318106 9734 raft_consensus.cc:2726] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 22:41:21.318135 9736 consensus_queue.cc:237] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200b7a862e10443ba89a5d209f4959aa" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44813 } } peers { permanent_uuid: "e74fb20181524e049aca671bdc2d4bbd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 37181 } } peers { permanent_uuid: "70289c55941042a9969b16ac4d442f7a" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38067 } }
I20260925 22:41:21.321854 8456 catalog_manager.cc:5495] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 70289c55941042a9969b16ac4d442f7a (127.2.74.194). New cstate: current_term: 1 leader_uuid: "70289c55941042a9969b16ac4d442f7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200b7a862e10443ba89a5d209f4959aa" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e74fb20181524e049aca671bdc2d4bbd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70289c55941042a9969b16ac4d442f7a" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38067 } health_report { overall_health: HEALTHY } } }
W20260925 22:41:21.331516 9561 tablet.cc:2369] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:41:21.336580 9727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:21.332894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44096 (local address 127.2.74.194:38067)
0925 22:41:21.333022 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:21.333025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:21.333040 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:41:21.333141 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:21.333143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:21.333196 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:21.333296 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:41:21.333510 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.334022 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.334849 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.335158 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.335247 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:21.335627 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:41:21.336104 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:41:21.336203 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:21.336312 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 22:41:21.336368 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:41:21.343055 9375 raft_consensus.cc:1261] T 463dee26894f4e99a190080b870962db P 200b7a862e10443ba89a5d209f4959aa [term 1 FOLLOWER]: Refusing update from remote peer 70289c55941042a9969b16ac4d442f7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:41:21.343055 9648 raft_consensus.cc:1261] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd [term 1 FOLLOWER]: Refusing update from remote peer 70289c55941042a9969b16ac4d442f7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:41:21.343834 9738 consensus_queue.cc:1035] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "200b7a862e10443ba89a5d209f4959aa" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:41:21.344048 9736 consensus_queue.cc:1035] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e74fb20181524e049aca671bdc2d4bbd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:41:21.354982 9742 mvcc.cc:205] Tried to move back new op lower bound from 7333380429173420032 to 7333380429081817088. Current Snapshot: MvccSnapshot[applied={T|T < 7333380429173420032}]
I20260925 22:41:21.362793 8457 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:41:21.363014 8457 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:41:21.367311 8457 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f44fda985f2248aa820cd857d80c5e3d: Sending DeleteTablet for 3 replicas of tablet 463dee26894f4e99a190080b870962db
I20260925 22:41:21.368198 9628 tablet_service.cc:1514] Processing DeleteTablet for tablet 463dee26894f4e99a190080b870962db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:21 UTC) from {username='oryx', principal='oryx/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:54778
I20260925 22:41:21.368417 9355 tablet_service.cc:1514] Processing DeleteTablet for tablet 463dee26894f4e99a190080b870962db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:21 UTC) from {username='oryx', principal='oryx/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:46792
I20260925 22:41:21.368839 9750 tablet_replica.cc:330] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd: stopping tablet replica
I20260925 22:41:21.368949 9751 tablet_replica.cc:330] T 463dee26894f4e99a190080b870962db P 200b7a862e10443ba89a5d209f4959aa: stopping tablet replica
I20260925 22:41:21.369129 9492 tablet_service.cc:1514] Processing DeleteTablet for tablet 463dee26894f4e99a190080b870962db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:21 UTC) from {username='oryx', principal='oryx/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:44088
I20260925 22:41:21.369310 9751 raft_consensus.cc:2229] T 463dee26894f4e99a190080b870962db P 200b7a862e10443ba89a5d209f4959aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:41:21.369359 9750 raft_consensus.cc:2229] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:41:21.369690 9751 raft_consensus.cc:2258] T 463dee26894f4e99a190080b870962db P 200b7a862e10443ba89a5d209f4959aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:41:21.369689 9750 raft_consensus.cc:2258] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:41:21.371376 9752 tablet_replica.cc:330] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a: stopping tablet replica
I20260925 22:41:21.371774 9752 raft_consensus.cc:2229] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:41:21.372196 9752 raft_consensus.cc:2258] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:41:21.373298 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 8426
I20260925 22:41:21.374173 9752 ts_tablet_manager.cc:1905] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:41:21.374779 9751 ts_tablet_manager.cc:1905] T 463dee26894f4e99a190080b870962db P 200b7a862e10443ba89a5d209f4959aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:41:21.375368 9750 ts_tablet_manager.cc:1905] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:41:21.377372 9752 ts_tablet_manager.cc:1918] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:41:21.377512 9752 log.cc:1196] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-1/wal/wals/463dee26894f4e99a190080b870962db
I20260925 22:41:21.377887 9752 ts_tablet_manager.cc:1939] T 463dee26894f4e99a190080b870962db P 70289c55941042a9969b16ac4d442f7a: Deleting consensus metadata
I20260925 22:41:21.380539 9751 ts_tablet_manager.cc:1918] T 463dee26894f4e99a190080b870962db P 200b7a862e10443ba89a5d209f4959aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:41:21.380682 9751 log.cc:1196] T 463dee26894f4e99a190080b870962db P 200b7a862e10443ba89a5d209f4959aa: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-0/wal/wals/463dee26894f4e99a190080b870962db
I20260925 22:41:21.381017 9751 ts_tablet_manager.cc:1939] T 463dee26894f4e99a190080b870962db P 200b7a862e10443ba89a5d209f4959aa: Deleting consensus metadata
W20260925 22:41:21.381503 9448 connection.cc:535] server connection from 127.0.0.1:44088 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 22:41:21.383204 9750 ts_tablet_manager.cc:1918] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:41:21.383327 9750 log.cc:1196] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/wal/wals/463dee26894f4e99a190080b870962db
I20260925 22:41:21.383695 9750 ts_tablet_manager.cc:1939] T 463dee26894f4e99a190080b870962db P e74fb20181524e049aca671bdc2d4bbd: Deleting consensus metadata
W20260925 22:41:21.385020 9581 connection.cc:441] server connection from 127.0.0.1:54778 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54778 (request call id 1) could send its response
W20260925 22:41:21.386032 9310 connection.cc:441] server connection from 127.0.0.1:46792 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46792 (request call id 1) could send its response
I20260925 22:41:21.396448 9705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:21.385517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50776 (local address 127.2.74.253:44125)
0925 22:41:21.385636 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:21.385640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:21.386149 (+ 509us) server_negotiation.cc:407] Connection header received
0925 22:41:21.386442 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:21.386445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:21.386494 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:21.386550 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:21.387590 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.388317 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.389576 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.389948 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.390033 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:21.391893 (+ 1860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:21.391897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:21.391900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:21.391945 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:21.394378 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:21.395012 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:21.395016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:21.395018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:21.395109 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:21.395427 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:21.395430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:21.395432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:21.395658 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:21.395769 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:21.396103 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 22:41:21.396244 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":38,"spinlock_wait_cycles":4224}
I20260925 22:41:21.396746 9182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:21.372359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50770 (local address 127.2.74.253:44125)
0925 22:41:21.372619 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:21.372623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:21.385296 (+ 12673us) server_negotiation.cc:407] Connection header received
0925 22:41:21.385522 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:21.385526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:21.385579 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:21.385632 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:21.386709 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.387552 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.388810 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.389155 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.389233 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:21.391883 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:21.391888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:21.391891 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:21.391936 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:21.394335 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:21.395051 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:21.395057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:21.395061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:21.395174 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:21.395514 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:21.395520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:21.395524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:21.395751 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:21.395867 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:21.396235 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 22:41:21.396351 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"spinlock_wait_cycles":7936}
I20260925 22:41:21.397656 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:21.386970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36914 (local address 127.2.74.252:43589)
0925 22:41:21.387114 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:21.387119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:21.387898 (+ 779us) server_negotiation.cc:407] Connection header received
0925 22:41:21.388061 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:21.388064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:21.388115 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:21.388163 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:21.389444 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.390215 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.391640 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.392006 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.392091 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:21.393898 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:21.393902 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:21.393906 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:21.393944 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:21.395803 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:21.396272 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:21.396275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:21.396276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:21.396357 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:21.396683 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:21.396686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:21.396688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:21.396882 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:21.396972 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:21.397332 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 22:41:21.397476 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:41:21.406318 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:21.372222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36908 (local address 127.2.74.252:43589)
0925 22:41:21.372513 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:21.372518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:21.397455 (+ 24937us) server_negotiation.cc:407] Connection header received
0925 22:41:21.397616 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:21.397619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:21.397663 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:21.397717 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:21.399103 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.399908 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.400804 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:21.401137 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:21.401247 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:21.402694 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:21.402701 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:21.402704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:21.402732 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:21.404597 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:21.405105 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:21.405109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:21.405111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:21.405198 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:21.405531 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:21.405534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:21.405535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:21.405726 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:21.405843 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:21.406100 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 22:41:21.406200 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260925 22:41:22.360286 9762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:22.359810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.2.74.194:58929)
0925 22:41:22.360086 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:22.360170 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":63,"threads_started":1}
I20260925 22:41:22.360366 9761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:22.359559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.2.74.195:52407)
0925 22:41:22.359876 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:22.360016 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":82,"threads_started":1}
W20260925 22:41:22.360737 9559 heartbeater.cc:644] Failed to heartbeat to 127.2.74.254:33523 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
I20260925 22:41:22.360849 9763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:22.360466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.2.74.193:45309)
0925 22:41:22.360669 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:22.360751 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":70,"threads_started":1}
W20260925 22:41:22.360841 9695 heartbeater.cc:644] Failed to heartbeat to 127.2.74.254:33523 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
W20260925 22:41:22.361318 9422 heartbeater.cc:644] Failed to heartbeat to 127.2.74.254:33523 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
I20260925 22:41:22.868449 9764 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f44fda985f2248aa820cd857d80c5e3d)
I20260925 22:41:22.868645 9764 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: false } }
I20260925 22:41:22.869706 9764 leader_election.cc:288] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f44fda985f2248aa820cd857d80c5e3d (127.2.74.254:33523), 8c135ec404cb42e7b809b5b70986d83b (127.2.74.252:43589)
I20260925 22:41:22.870410 9765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:22.869938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.0.0.1:52658)
0925 22:41:22.870183 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:22.870272 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":95,"threads_started":1}
W20260925 22:41:22.870724 9046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
I20260925 22:41:22.870898 9768 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f44fda985f2248aa820cd857d80c5e3d)
I20260925 22:41:22.871130 9768 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: false } }
I20260925 22:41:22.871809 9765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:22.871562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.0.0.1:52668)
0925 22:41:22.871688 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:22.871744 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 22:41:22.871978 9046 leader_election.cc:334] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f44fda985f2248aa820cd857d80c5e3d (127.2.74.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
I20260925 22:41:22.872299 9768 leader_election.cc:288] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f44fda985f2248aa820cd857d80c5e3d (127.2.74.254:33523), fbdb039a1ac44df999d6b05de7cb0f77 (127.2.74.253:44125)
I20260925 22:41:22.872978 9770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:22.872345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.0.0.1:52682)
0925 22:41:22.872823 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:22.872890 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":67,"threads_started":1}
W20260925 22:41:22.873142 9228 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
I20260925 22:41:22.873965 9771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:22.873708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.0.0.1:52684)
0925 22:41:22.873809 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:22.873871 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 22:41:22.874070 9228 leader_election.cc:334] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f44fda985f2248aa820cd857d80c5e3d (127.2.74.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
I20260925 22:41:22.874792 9767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:22.870554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36928 (local address 127.2.74.252:43589)
0925 22:41:22.870846 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:22.870850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:22.870867 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:41:22.870924 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:22.870927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:22.871052 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:22.871205 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:22.871441 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:22.872202 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:22.873739 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:22.874310 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:22.874406 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:22.874454 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:22.874644 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 22:41:22.874699 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":168,"thread_start_us":76,"threads_started":1}
I20260925 22:41:22.874830 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:22.870307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:43589 (local address 127.0.0.1:36928)
0925 22:41:22.870525 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:22.870540 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:22.870658 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:22.871217 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:22.871220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:22.871239 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:22.871345 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:22.871352 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:22.872325 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:22.872331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:22.873613 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:22.873623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:22.874432 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:22.874435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:22.874525 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:22.874551 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 22:41:22.874634 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 22:41:22.874701 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":64,"threads_started":1}
I20260925 22:41:22.875341 9244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8c135ec404cb42e7b809b5b70986d83b" is_pre_election: true
I20260925 22:41:22.875551 9244 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbdb039a1ac44df999d6b05de7cb0f77 in term 1.
I20260925 22:41:22.875913 9046 leader_election.cc:302] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c135ec404cb42e7b809b5b70986d83b, fbdb039a1ac44df999d6b05de7cb0f77; no voters: f44fda985f2248aa820cd857d80c5e3d
I20260925 22:41:22.876168 9764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 22:41:22.876236 9764 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f44fda985f2248aa820cd857d80c5e3d)
I20260925 22:41:22.876287 9764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:41:22.877283 9770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:22.872484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:44125 (local address 127.0.0.1:50792)
0925 22:41:22.873521 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:22.873534 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:22.873628 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:22.873964 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:22.873967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:22.873989 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:22.874094 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:22.874100 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:22.875051 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:22.875054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:22.876082 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:22.876094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:22.876965 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:22.876968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:22.877037 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:22.877057 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 22:41:22.877130 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 22:41:22.877185 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":987,"mutex_wait_us":4,"thread_start_us":53,"threads_started":1}
I20260925 22:41:22.877362 9772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:22.872742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50792 (local address 127.2.74.253:44125)
0925 22:41:22.872986 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:22.872992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:22.873710 (+ 718us) server_negotiation.cc:407] Connection header received
0925 22:41:22.873751 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:22.873754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:22.873794 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:22.873865 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:22.874219 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:22.874938 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:22.876219 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:22.876809 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:22.876929 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:22.876989 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:22.877164 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 22:41:22.877247 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":103,"threads_started":1}
I20260925 22:41:22.877630 9764 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: false } }
I20260925 22:41:22.877956 9764 leader_election.cc:288] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [CANDIDATE]: Term 2 election: Requested vote from peers f44fda985f2248aa820cd857d80c5e3d (127.2.74.254:33523), 8c135ec404cb42e7b809b5b70986d83b (127.2.74.252:43589)
I20260925 22:41:22.878373 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:22.878115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.0.0.1:52700)
0925 22:41:22.878223 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:22.878309 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 22:41:22.878443 9070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c135ec404cb42e7b809b5b70986d83b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" is_pre_election: true
I20260925 22:41:22.878502 9244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8c135ec404cb42e7b809b5b70986d83b"
I20260925 22:41:22.878612 9244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:41:22.878664 9070 raft_consensus.cc:2379] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c135ec404cb42e7b809b5b70986d83b in current term 2: Already voted for candidate fbdb039a1ac44df999d6b05de7cb0f77 in this term.
I20260925 22:41:22.879087 9226 leader_election.cc:302] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c135ec404cb42e7b809b5b70986d83b; no voters: f44fda985f2248aa820cd857d80c5e3d, fbdb039a1ac44df999d6b05de7cb0f77
I20260925 22:41:22.879153 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:22.878850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.0.0.1:52712)
0925 22:41:22.879041 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:22.879093 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260925 22:41:22.879380 9046 leader_election.cc:334] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f44fda985f2248aa820cd857d80c5e3d (127.2.74.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
I20260925 22:41:22.879832 9244 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbdb039a1ac44df999d6b05de7cb0f77 in term 2.
I20260925 22:41:22.879983 9768 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:41:22.880168 9046 leader_election.cc:302] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c135ec404cb42e7b809b5b70986d83b, fbdb039a1ac44df999d6b05de7cb0f77; no voters: f44fda985f2248aa820cd857d80c5e3d
I20260925 22:41:22.880327 9764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 22:41:22.880513 9764 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [term 2 LEADER]: Becoming Leader. State: Replica: fbdb039a1ac44df999d6b05de7cb0f77, State: Running, Role: LEADER
I20260925 22:41:22.880770 9764 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: false } }
I20260925 22:41:22.882093 9774 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbdb039a1ac44df999d6b05de7cb0f77. Latest consensus state: current_term: 2 leader_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: false } } }
I20260925 22:41:22.882233 9774 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:22.882582 9776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:41:22.885146 9776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:41:22.885488 9776 catalog_manager.cc:1259] Loaded cluster ID: 0747fe35ece54cc8b064a2929f65b7e5
I20260925 22:41:22.885528 9776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:41:22.886005 9776 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:41:22.886346 9776 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77: Loaded TSK: 0
I20260925 22:41:22.887040 9776 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:41:22.926424 9058 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:41:22.929705 9058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:41:22.933151 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:22.932830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.0.0.1:52728)
0925 22:41:22.932965 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:22.933055 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 22:41:22.933466 9244 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [term 2 FOLLOWER]: Refusing update from remote peer fbdb039a1ac44df999d6b05de7cb0f77: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260925 22:41:22.933923 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:22.933686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.0.0.1:52730)
0925 22:41:22.933813 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:22.933863 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 22:41:22.934020 9764 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260925 22:41:22.934276 9046 consensus_peers.cc:487] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 -> Peer f44fda985f2248aa820cd857d80c5e3d (127.2.74.254:33523): Couldn't send request to peer f44fda985f2248aa820cd857d80c5e3d. Status: Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 22:41:22.935717 9768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbdb039a1ac44df999d6b05de7cb0f77. Latest consensus state: current_term: 2 leader_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: false } } }
I20260925 22:41:22.935871 9768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:41:22.937537 9768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: false } } }
I20260925 22:41:22.937670 9768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c135ec404cb42e7b809b5b70986d83b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:41:22.938561 9782 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f44fda985f2248aa820cd857d80c5e3d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33523 } } peers { permanent_uuid: "fbdb039a1ac44df999d6b05de7cb0f77" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 44125 } attrs { promote: false } } peers { permanent_uuid: "8c135ec404cb42e7b809b5b70986d83b" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 43589 } attrs { promote: false } } }
I20260925 22:41:22.938697 9782 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77 [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:22.948660 9783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:22.944350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54788 (local address 127.2.74.195:37181)
0925 22:41:22.944949 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:22.944954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:22.944969 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:41:22.945035 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:22.945038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:22.945102 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:22.945189 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:22.945422 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:22.946078 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:22.947506 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:22.948087 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:22.948202 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:22.948271 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:22.948439 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 22:41:22.948512 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":85,"threads_started":1}
I20260925 22:41:22.948659 9765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:22.944121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:37181 (local address 127.0.0.1:54788)
0925 22:41:22.944626 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:22.944643 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:22.944729 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:22.945201 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:22.945203 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:22.945217 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:22.945306 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:22.945312 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:22.946204 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:22.946207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:22.947360 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:22.947369 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:22.948202 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:22.948206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:22.948297 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:22.948319 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:41:22.948413 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 22:41:22.948505 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":4,"spinlock_wait_cycles":384}
I20260925 22:41:22.949155 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:22.944464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:38067 (local address 127.0.0.1:44106)
0925 22:41:22.944626 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:22.944643 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:22.944734 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:22.945134 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:22.945137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:22.945154 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:22.945284 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:22.945291 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:22.946484 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:22.946487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:22.947834 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:22.947843 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:22.948754 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:22.948758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:22.948854 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:22.948878 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:41:22.948925 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 22:41:22.949010 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":54,"spinlock_wait_cycles":1792}
I20260925 22:41:22.949172 9785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:22.944705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:44813 (local address 127.0.0.1:46796)
0925 22:41:22.944953 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:22.944966 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:22.945042 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:22.945558 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:22.945561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:22.945578 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:22.945651 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:22.945656 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:22.946618 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:22.946621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:22.947842 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:22.947851 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:22.948842 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:22.948845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:22.948928 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:22.948946 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 22:41:22.949012 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 22:41:22.949069 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":23,"thread_start_us":108,"threads_started":1}
I20260925 22:41:22.949318 9628 tablet_service.cc:1467] Processing CreateTablet for tablet 1acf317886814a56bbaab3a16a17a0eb (DEFAULT_TABLE table=test-table [id=5882057a202248cdb5393b1e861e2467]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:22.949388 9786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:22.945066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46796 (local address 127.2.74.193:44813)
0925 22:41:22.945337 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:22.945343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:22.945358 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:41:22.945410 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:22.945412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:22.945457 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:22.945543 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:22.945771 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:22.946496 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:22.948001 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:22.948613 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:22.948729 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:22.948794 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:22.949054 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 22:41:22.949148 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":95,"threads_started":1}
I20260925 22:41:22.949630 9628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1acf317886814a56bbaab3a16a17a0eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:22.949895 9355 tablet_service.cc:1467] Processing CreateTablet for tablet 1acf317886814a56bbaab3a16a17a0eb (DEFAULT_TABLE table=test-table [id=5882057a202248cdb5393b1e861e2467]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:22.950169 9355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1acf317886814a56bbaab3a16a17a0eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:22.950420 9784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:22.944572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44106 (local address 127.2.74.194:38067)
0925 22:41:22.944838 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:22.944843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:22.944857 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:41:22.944922 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:22.944926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:22.944980 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:22.945419 (+ 439us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:22.945553 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:22.946360 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:22.947960 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:22.948609 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:22.950106 (+ 1497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:22.950162 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:22.950195 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 22:41:22.950307 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":61,"threads_started":1}
I20260925 22:41:22.950881 9492 tablet_service.cc:1467] Processing CreateTablet for tablet 1acf317886814a56bbaab3a16a17a0eb (DEFAULT_TABLE table=test-table [id=5882057a202248cdb5393b1e861e2467]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:22.951187 9492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1acf317886814a56bbaab3a16a17a0eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:22.952476 9787 tablet_bootstrap.cc:492] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd: Bootstrap starting.
I20260925 22:41:22.953469 9789 tablet_bootstrap.cc:492] T 1acf317886814a56bbaab3a16a17a0eb P 70289c55941042a9969b16ac4d442f7a: Bootstrap starting.
I20260925 22:41:22.953578 9788 tablet_bootstrap.cc:492] T 1acf317886814a56bbaab3a16a17a0eb P 200b7a862e10443ba89a5d209f4959aa: Bootstrap starting.
I20260925 22:41:22.953612 9787 tablet_bootstrap.cc:654] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:22.954394 9789 tablet_bootstrap.cc:654] T 1acf317886814a56bbaab3a16a17a0eb P 70289c55941042a9969b16ac4d442f7a: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:22.954394 9788 tablet_bootstrap.cc:654] T 1acf317886814a56bbaab3a16a17a0eb P 200b7a862e10443ba89a5d209f4959aa: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:22.955128 9787 tablet_bootstrap.cc:492] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd: No bootstrap required, opened a new log
I20260925 22:41:22.955242 9787 ts_tablet_manager.cc:1397] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:41:22.955665 9788 tablet_bootstrap.cc:492] T 1acf317886814a56bbaab3a16a17a0eb P 200b7a862e10443ba89a5d209f4959aa: No bootstrap required, opened a new log
I20260925 22:41:22.955621 9787 raft_consensus.cc:348] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74fb20181524e049aca671bdc2d4bbd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 37181 } } peers { permanent_uuid: "70289c55941042a9969b16ac4d442f7a" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38067 } } peers { permanent_uuid: "200b7a862e10443ba89a5d209f4959aa" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44813 } }
I20260925 22:41:22.955678 9789 tablet_bootstrap.cc:492] T 1acf317886814a56bbaab3a16a17a0eb P 70289c55941042a9969b16ac4d442f7a: No bootstrap required, opened a new log
I20260925 22:41:22.955724 9787 raft_consensus.cc:374] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:22.955745 9787 raft_consensus.cc:729] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e74fb20181524e049aca671bdc2d4bbd, State: Initialized, Role: FOLLOWER
I20260925 22:41:22.955775 9788 ts_tablet_manager.cc:1397] T 1acf317886814a56bbaab3a16a17a0eb P 200b7a862e10443ba89a5d209f4959aa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 22:41:22.955891 9787 consensus_queue.cc:260] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74fb20181524e049aca671bdc2d4bbd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 37181 } } peers { permanent_uuid: "70289c55941042a9969b16ac4d442f7a" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38067 } } peers { permanent_uuid: "200b7a862e10443ba89a5d209f4959aa" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44813 } }
I20260925 22:41:22.956176 9789 ts_tablet_manager.cc:1397] T 1acf317886814a56bbaab3a16a17a0eb P 70289c55941042a9969b16ac4d442f7a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:41:22.956234 9787 ts_tablet_manager.cc:1428] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:41:22.956238 9788 raft_consensus.cc:348] T 1acf317886814a56bbaab3a16a17a0eb P 200b7a862e10443ba89a5d209f4959aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74fb20181524e049aca671bdc2d4bbd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 37181 } } peers { permanent_uuid: "70289c55941042a9969b16ac4d442f7a" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38067 } } peers { permanent_uuid: "200b7a862e10443ba89a5d209f4959aa" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44813 } }
I20260925 22:41:22.956362 9788 raft_consensus.cc:374] T 1acf317886814a56bbaab3a16a17a0eb P 200b7a862e10443ba89a5d209f4959aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:22.956399 9788 raft_consensus.cc:729] T 1acf317886814a56bbaab3a16a17a0eb P 200b7a862e10443ba89a5d209f4959aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 200b7a862e10443ba89a5d209f4959aa, State: Initialized, Role: FOLLOWER
I20260925 22:41:22.956542 9788 consensus_queue.cc:260] T 1acf317886814a56bbaab3a16a17a0eb P 200b7a862e10443ba89a5d209f4959aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74fb20181524e049aca671bdc2d4bbd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 37181 } } peers { permanent_uuid: "70289c55941042a9969b16ac4d442f7a" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38067 } } peers { permanent_uuid: "200b7a862e10443ba89a5d209f4959aa" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44813 } }
I20260925 22:41:22.956885 9788 ts_tablet_manager.cc:1428] T 1acf317886814a56bbaab3a16a17a0eb P 200b7a862e10443ba89a5d209f4959aa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:41:22.956907 9789 raft_consensus.cc:348] T 1acf317886814a56bbaab3a16a17a0eb P 70289c55941042a9969b16ac4d442f7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74fb20181524e049aca671bdc2d4bbd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 37181 } } peers { permanent_uuid: "70289c55941042a9969b16ac4d442f7a" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38067 } } peers { permanent_uuid: "200b7a862e10443ba89a5d209f4959aa" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44813 } }
I20260925 22:41:22.957115 9789 raft_consensus.cc:374] T 1acf317886814a56bbaab3a16a17a0eb P 70289c55941042a9969b16ac4d442f7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:22.957192 9789 raft_consensus.cc:729] T 1acf317886814a56bbaab3a16a17a0eb P 70289c55941042a9969b16ac4d442f7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70289c55941042a9969b16ac4d442f7a, State: Initialized, Role: FOLLOWER
I20260925 22:41:22.957404 9789 consensus_queue.cc:260] T 1acf317886814a56bbaab3a16a17a0eb P 70289c55941042a9969b16ac4d442f7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74fb20181524e049aca671bdc2d4bbd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 37181 } } peers { permanent_uuid: "70289c55941042a9969b16ac4d442f7a" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38067 } } peers { permanent_uuid: "200b7a862e10443ba89a5d209f4959aa" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44813 } }
I20260925 22:41:22.957808 9789 ts_tablet_manager.cc:1428] T 1acf317886814a56bbaab3a16a17a0eb P 70289c55941042a9969b16ac4d442f7a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 22:41:22.958695 9795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:22.956700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.2.74.195:50613)
0925 22:41:22.958497 (+ 1797us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:22.958577 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1689,"thread_start_us":99,"threads_started":1}
I20260925 22:41:22.959703 9795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:22.959136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.2.74.195:46085)
0925 22:41:22.959550 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:22.959619 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":93,"threads_started":1}
I20260925 22:41:22.960289 9797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:22.958714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.2.74.194:56545)
0925 22:41:22.960117 (+ 1403us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:22.960181 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1339,"thread_start_us":113,"threads_started":1}
I20260925 22:41:22.960835 9799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:22.960401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.2.74.193:45285)
0925 22:41:22.960674 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:22.960734 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":107,"threads_started":1}
I20260925 22:41:22.960844 9423 heartbeater.cc:503] Master 127.2.74.253:44125 was elected leader, sending a full tablet report...
I20260925 22:41:22.961298 9696 heartbeater.cc:503] Master 127.2.74.253:44125 was elected leader, sending a full tablet report...
I20260925 22:41:22.962327 9799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:22.961953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.2.74.193:42891)
0925 22:41:22.962200 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:22.962251 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260925 22:41:22.963616 9560 heartbeater.cc:503] Master 127.2.74.253:44125 was elected leader, sending a full tablet report...
I20260925 22:41:22.964099 9797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:22.963715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.2.74.194:44475)
0925 22:41:22.963956 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:22.964022 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260925 22:41:23.090525 9793 raft_consensus.cc:482] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:41:23.090701 9793 raft_consensus.cc:504] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74fb20181524e049aca671bdc2d4bbd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 37181 } } peers { permanent_uuid: "70289c55941042a9969b16ac4d442f7a" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38067 } } peers { permanent_uuid: "200b7a862e10443ba89a5d209f4959aa" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44813 } }
I20260925 22:41:23.091171 9793 leader_election.cc:288] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70289c55941042a9969b16ac4d442f7a (127.2.74.194:38067), 200b7a862e10443ba89a5d209f4959aa (127.2.74.193:44813)
I20260925 22:41:23.091778 9375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1acf317886814a56bbaab3a16a17a0eb" candidate_uuid: "e74fb20181524e049aca671bdc2d4bbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "200b7a862e10443ba89a5d209f4959aa" is_pre_election: true
I20260925 22:41:23.091777 9512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1acf317886814a56bbaab3a16a17a0eb" candidate_uuid: "e74fb20181524e049aca671bdc2d4bbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70289c55941042a9969b16ac4d442f7a" is_pre_election: true
I20260925 22:41:23.091948 9512 raft_consensus.cc:2454] T 1acf317886814a56bbaab3a16a17a0eb P 70289c55941042a9969b16ac4d442f7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e74fb20181524e049aca671bdc2d4bbd in term 0.
I20260925 22:41:23.091948 9375 raft_consensus.cc:2454] T 1acf317886814a56bbaab3a16a17a0eb P 200b7a862e10443ba89a5d209f4959aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e74fb20181524e049aca671bdc2d4bbd in term 0.
I20260925 22:41:23.092348 9584 leader_election.cc:302] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 200b7a862e10443ba89a5d209f4959aa, 70289c55941042a9969b16ac4d442f7a, e74fb20181524e049aca671bdc2d4bbd; no voters:
I20260925 22:41:23.092535 9793 raft_consensus.cc:2781] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:41:23.092607 9793 raft_consensus.cc:482] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:41:23.092638 9793 raft_consensus.cc:3037] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:23.093600 9793 raft_consensus.cc:504] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74fb20181524e049aca671bdc2d4bbd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 37181 } } peers { permanent_uuid: "70289c55941042a9969b16ac4d442f7a" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38067 } } peers { permanent_uuid: "200b7a862e10443ba89a5d209f4959aa" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44813 } }
I20260925 22:41:23.093969 9793 leader_election.cc:288] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd [CANDIDATE]: Term 1 election: Requested vote from peers 70289c55941042a9969b16ac4d442f7a (127.2.74.194:38067), 200b7a862e10443ba89a5d209f4959aa (127.2.74.193:44813)
I20260925 22:41:23.094377 9512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1acf317886814a56bbaab3a16a17a0eb" candidate_uuid: "e74fb20181524e049aca671bdc2d4bbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70289c55941042a9969b16ac4d442f7a"
I20260925 22:41:23.094377 9375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1acf317886814a56bbaab3a16a17a0eb" candidate_uuid: "e74fb20181524e049aca671bdc2d4bbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "200b7a862e10443ba89a5d209f4959aa"
I20260925 22:41:23.094516 9375 raft_consensus.cc:3037] T 1acf317886814a56bbaab3a16a17a0eb P 200b7a862e10443ba89a5d209f4959aa [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:23.094532 9512 raft_consensus.cc:3037] T 1acf317886814a56bbaab3a16a17a0eb P 70289c55941042a9969b16ac4d442f7a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:23.095736 9375 raft_consensus.cc:2454] T 1acf317886814a56bbaab3a16a17a0eb P 200b7a862e10443ba89a5d209f4959aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e74fb20181524e049aca671bdc2d4bbd in term 1.
I20260925 22:41:23.095736 9512 raft_consensus.cc:2454] T 1acf317886814a56bbaab3a16a17a0eb P 70289c55941042a9969b16ac4d442f7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e74fb20181524e049aca671bdc2d4bbd in term 1.
I20260925 22:41:23.096077 9583 leader_election.cc:302] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70289c55941042a9969b16ac4d442f7a, e74fb20181524e049aca671bdc2d4bbd; no voters:
I20260925 22:41:23.096256 9793 raft_consensus.cc:2781] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:41:23.096508 9793 raft_consensus.cc:686] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd [term 1 LEADER]: Becoming Leader. State: Replica: e74fb20181524e049aca671bdc2d4bbd, State: Running, Role: LEADER
I20260925 22:41:23.096889 9793 consensus_queue.cc:237] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74fb20181524e049aca671bdc2d4bbd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 37181 } } peers { permanent_uuid: "70289c55941042a9969b16ac4d442f7a" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38067 } } peers { permanent_uuid: "200b7a862e10443ba89a5d209f4959aa" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44813 } }
I20260925 22:41:23.097473 9795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:23.097164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.2.74.195:47027)
0925 22:41:23.097300 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:23.097372 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 22:41:23.099447 9058 catalog_manager.cc:5495] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd reported cstate change: term changed from 0 to 1, leader changed from <none> to e74fb20181524e049aca671bdc2d4bbd (127.2.74.195). New cstate: current_term: 1 leader_uuid: "e74fb20181524e049aca671bdc2d4bbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74fb20181524e049aca671bdc2d4bbd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 37181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70289c55941042a9969b16ac4d442f7a" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "200b7a862e10443ba89a5d209f4959aa" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44813 } health_report { overall_health: UNKNOWN } } }
I20260925 22:41:23.142774 9783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:23.139510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54798 (local address 127.2.74.195:37181)
0925 22:41:23.139639 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:23.139643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:23.139655 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:41:23.139729 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:23.139731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:23.139787 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:23.139879 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:41:23.140085 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:23.140606 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:23.141405 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:23.141614 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:23.141690 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:23.141973 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:41:23.142446 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:41:23.142546 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:23.142606 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 22:41:23.142660 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:41:23.145599 9375 raft_consensus.cc:1261] T 1acf317886814a56bbaab3a16a17a0eb P 200b7a862e10443ba89a5d209f4959aa [term 1 FOLLOWER]: Refusing update from remote peer e74fb20181524e049aca671bdc2d4bbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:41:23.145821 9512 raft_consensus.cc:1261] T 1acf317886814a56bbaab3a16a17a0eb P 70289c55941042a9969b16ac4d442f7a [term 1 FOLLOWER]: Refusing update from remote peer e74fb20181524e049aca671bdc2d4bbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:41:23.146133 9793 consensus_queue.cc:1035] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "200b7a862e10443ba89a5d209f4959aa" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:41:23.146486 9800 consensus_queue.cc:1035] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "70289c55941042a9969b16ac4d442f7a" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:41:23.148299 9799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:23.147984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.2.74.193:40085)
0925 22:41:23.148143 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:23.148211 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 22:41:23.149600 9803 mvcc.cc:205] Tried to move back new op lower bound from 7333380436559523840 to 7333380436367917056. Current Snapshot: MvccSnapshot[applied={T|T < 7333380436559523840}]
I20260925 22:41:23.149820 9797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:23.149297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.2.74.194:49045)
0925 22:41:23.149680 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:23.149744 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304}
I20260925 22:41:23.151316 9795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:23.150910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.2.74.195:53809)
0925 22:41:23.151183 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:23.151242 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
I20260925 22:41:23.152141 9799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:23.151811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.2.74.193:44153)
0925 22:41:23.152021 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:23.152072 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260925 22:41:23.154740 9806 mvcc.cc:205] Tried to move back new op lower bound from 7333380436559523840 to 7333380436367917056. Current Snapshot: MvccSnapshot[applied={T|T < 7333380436559523840}]
I20260925 22:41:23.155498 9797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:23.155143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.2.74.194:42959)
0925 22:41:23.155365 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:23.155422 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260925 22:41:23.155794 9795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:23.154173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33523 (local address 127.2.74.195:55381)
0925 22:41:23.155669 (+ 1496us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:23.155721 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:33523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1430}
I20260925 22:41:23.161530 9058 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:41:23.161707 9058 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:41:23.165905 9058 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fbdb039a1ac44df999d6b05de7cb0f77: Sending DeleteTablet for 3 replicas of tablet 1acf317886814a56bbaab3a16a17a0eb
I20260925 22:41:23.166805 9492 tablet_service.cc:1514] Processing DeleteTablet for tablet 1acf317886814a56bbaab3a16a17a0eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:23 UTC) from {username='oryx', principal='oryx/127.2.74.253@KRBTEST.COM'} at 127.0.0.1:44106
I20260925 22:41:23.167027 9355 tablet_service.cc:1514] Processing DeleteTablet for tablet 1acf317886814a56bbaab3a16a17a0eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:23 UTC) from {username='oryx', principal='oryx/127.2.74.253@KRBTEST.COM'} at 127.0.0.1:46796
I20260925 22:41:23.167049 9628 tablet_service.cc:1514] Processing DeleteTablet for tablet 1acf317886814a56bbaab3a16a17a0eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:23 UTC) from {username='oryx', principal='oryx/127.2.74.253@KRBTEST.COM'} at 127.0.0.1:54788
I20260925 22:41:23.167577 9813 tablet_replica.cc:330] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd: stopping tablet replica
I20260925 22:41:23.167762 9813 raft_consensus.cc:2229] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:41:23.168100 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 9204
I20260925 22:41:23.170861 9813 raft_consensus.cc:2258] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:41:23.171746 9813 ts_tablet_manager.cc:1905] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:41:23.174698 9813 ts_tablet_manager.cc:1918] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:41:23.174814 9813 log.cc:1196] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376047580784-2347-0/minicluster-data/ts-2/wal/wals/1acf317886814a56bbaab3a16a17a0eb
I20260925 22:41:23.175156 9813 ts_tablet_manager.cc:1939] T 1acf317886814a56bbaab3a16a17a0eb P e74fb20181524e049aca671bdc2d4bbd: Deleting consensus metadata
I20260925 22:41:23.175320 9814 tablet_replica.cc:330] T 1acf317886814a56bbaab3a16a17a0eb P 70289c55941042a9969b16ac4d442f7a: stopping tablet replica
I20260925 22:41:23.175890 9814 raft_consensus.cc:2229] T 1acf317886814a56bbaab3a16a17a0eb P 70289c55941042a9969b16ac4d442f7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:41:23.175967 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 9429
I20260925 22:41:23.177299 9785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:23.176939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:44813 (local address 127.0.0.1:46810)
0925 22:41:23.177149 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:23.177203 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.193:44813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260925 22:41:23.177518 9047 catalog_manager.cc:4584] TS 200b7a862e10443ba89a5d209f4959aa (127.2.74.193:44813): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1acf317886814a56bbaab3a16a17a0eb: Network error: Client connection negotiation failed: client connection to 127.2.74.193:44813: connect: Connection refused (error 111)
I20260925 22:41:23.178356 9047 catalog_manager.cc:4841] TS e74fb20181524e049aca671bdc2d4bbd (127.2.74.195:37181): tablet 1acf317886814a56bbaab3a16a17a0eb (table test-table [id=5882057a202248cdb5393b1e861e2467]) successfully deleted
I20260925 22:41:23.183537 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 9565
I20260925 22:41:23.183687 9785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:23.183361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:38067 (local address 127.0.0.1:44122)
0925 22:41:23.183541 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:23.183594 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.194:38067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260925 22:41:23.189841 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 8922
I20260925 22:41:23.196111 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 9024
2026-09-25T22:41:23Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5566 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:41:23 dist-test-slave-csnl krb5kdc[9817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:41:23 dist-test-slave-csnl krb5kdc[9817](info): set up 2 sockets
Sep 25 22:41:23 dist-test-slave-csnl krb5kdc[9817](info): commencing operation
krb5kdc: starting...
W20260925 22:41:24.231814 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:41:24 dist-test-slave-csnl krb5kdc[9817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:41:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:41:24Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.254.keytab.
Entry for principal oryx/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.254.keytab.
I20260925 22:41:24.358559 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:38925
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.254.keytab
--principal=oryx/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:38737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:38925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:24.474246 9830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:24.474551 9830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:24.474602 9830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:24.478811 9830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:41:24.478900 9830 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 22:41:24.478919 9830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:24.478938 9830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:24.479043 9830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:41:24.479066 9830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:41:24.484797 9830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:38737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:38925
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.254.keytab
--principal=oryx/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:38925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:24.485901 9830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:24.486922 9830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:24.496582 9836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:24.496601 9830 server_base.cc:1034] running on GCE node
W20260925 22:41:24.496622 9838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:24.496582 9835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:24.497509 9830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:24.498771 9830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:24.499976 9830 hybrid_clock.cc:648] HybridClock initialized: now 1790376084499961 us; error 39 us; skew 500 ppm
Sep 25 22:41:24 dist-test-slave-csnl krb5kdc[9817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376084, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:24.502812 9830 init.cc:377] Logged in from keytab as oryx/127.2.74.254@KRBTEST.COM (short username oryx)
I20260925 22:41:24.503789 9830 webserver.cc:460] Webserver started at http://127.2.74.254:42781/ using document root <none> and password file <none>
I20260925 22:41:24.504360 9830 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:24.504422 9830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:24.504657 9830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:24.506812 9830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "8178a160028d4173af41c7df23c6947b"
format_stamp: "Formatted at 2026-09-25 22:41:24 on dist-test-slave-csnl"
server_key: "300aff3006befdc339553c9e8e6e1cf3"
server_key_iv: "684e3f247621def34ee16e51c3444e2e"
server_key_version: "encryptionkey@0"
I20260925 22:41:24.507436 9830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "8178a160028d4173af41c7df23c6947b"
format_stamp: "Formatted at 2026-09-25 22:41:24 on dist-test-slave-csnl"
server_key: "300aff3006befdc339553c9e8e6e1cf3"
server_key_iv: "684e3f247621def34ee16e51c3444e2e"
server_key_version: "encryptionkey@0"
I20260925 22:41:24.511435 9830 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:41:24.513962 9845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:24.515441 9830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:41:24.515588 9830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "8178a160028d4173af41c7df23c6947b"
format_stamp: "Formatted at 2026-09-25 22:41:24 on dist-test-slave-csnl"
server_key: "300aff3006befdc339553c9e8e6e1cf3"
server_key_iv: "684e3f247621def34ee16e51c3444e2e"
server_key_version: "encryptionkey@0"
I20260925 22:41:24.515717 9830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:24.544562 9830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:24.547874 9830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:24.548116 9830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:24.556504 9830 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:38925
I20260925 22:41:24.556529 9907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:38925 every 8 connection(s)
I20260925 22:41:24.557538 9830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:41:24.560401 9908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:24.565692 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 9830
I20260925 22:41:24.565800 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:41:24.566087 2347 external_mini_cluster.cc:1428] Setting key 1a20d51a2c94d7e9137f16b4a44436d9
I20260925 22:41:24.566286 9908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8178a160028d4173af41c7df23c6947b: Bootstrap starting.
I20260925 22:41:24.568964 9908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8178a160028d4173af41c7df23c6947b: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:24.569772 9908 log.cc:824] T 00000000000000000000000000000000 P 8178a160028d4173af41c7df23c6947b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:24.572252 9908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8178a160028d4173af41c7df23c6947b: No bootstrap required, opened a new log
Sep 25 22:41:24 dist-test-slave-csnl krb5kdc[9817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM
I20260925 22:41:24.577035 9908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8178a160028d4173af41c7df23c6947b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8178a160028d4173af41c7df23c6947b" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38925 } }
I20260925 22:41:24.577387 9908 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8178a160028d4173af41c7df23c6947b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:24.577461 9908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8178a160028d4173af41c7df23c6947b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8178a160028d4173af41c7df23c6947b, State: Initialized, Role: FOLLOWER
I20260925 22:41:24.578027 9908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8178a160028d4173af41c7df23c6947b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8178a160028d4173af41c7df23c6947b" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38925 } }
I20260925 22:41:24.578202 9908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8178a160028d4173af41c7df23c6947b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:41:24.578299 9908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8178a160028d4173af41c7df23c6947b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:41:24.578411 9908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8178a160028d4173af41c7df23c6947b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:24.579650 9908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8178a160028d4173af41c7df23c6947b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8178a160028d4173af41c7df23c6947b" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38925 } }
I20260925 22:41:24.580075 9908 leader_election.cc:302] T 00000000000000000000000000000000 P 8178a160028d4173af41c7df23c6947b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8178a160028d4173af41c7df23c6947b; no voters:
I20260925 22:41:24.580416 9908 leader_election.cc:288] T 00000000000000000000000000000000 P 8178a160028d4173af41c7df23c6947b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:41:24.580538 9913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8178a160028d4173af41c7df23c6947b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:41:24.580775 9913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8178a160028d4173af41c7df23c6947b [term 1 LEADER]: Becoming Leader. State: Replica: 8178a160028d4173af41c7df23c6947b, State: Running, Role: LEADER
I20260925 22:41:24.581143 9913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8178a160028d4173af41c7df23c6947b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8178a160028d4173af41c7df23c6947b" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38925 } }
I20260925 22:41:24.581786 9908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8178a160028d4173af41c7df23c6947b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:41:24.582680 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:24.567531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42822 (local address 127.2.74.254:38925)
0925 22:41:24.568050 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:24.568066 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:24.568103 (+ 37us) server_negotiation.cc:407] Connection header received
0925 22:41:24.568790 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:24.568809 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:24.569241 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:24.569630 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:24.570549 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:24.572041 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:24.573090 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:24.573786 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:24.573973 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:24.576735 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:24.576742 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:24.576760 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:24.576798 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:24.578879 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:24.579523 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:24.579528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:24.579534 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:24.579668 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:24.579988 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:24.579991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:24.579992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:24.580570 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:24.580787 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:24.581266 (+ 479us) server_negotiation.cc:299] Negotiation successful
0925 22:41:24.581627 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":160,"threads_started":1}
I20260925 22:41:24.582880 9914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8178a160028d4173af41c7df23c6947b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8178a160028d4173af41c7df23c6947b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8178a160028d4173af41c7df23c6947b" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38925 } } }
I20260925 22:41:24.583037 9914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8178a160028d4173af41c7df23c6947b [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:24.583289 9914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8178a160028d4173af41c7df23c6947b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8178a160028d4173af41c7df23c6947b. Latest consensus state: current_term: 1 leader_uuid: "8178a160028d4173af41c7df23c6947b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8178a160028d4173af41c7df23c6947b" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38925 } } }
I20260925 22:41:24.583380 9914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8178a160028d4173af41c7df23c6947b [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:24.583675 9922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:41:24.587683 9922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:41:24.594236 9922 catalog_manager.cc:1347] Generated new cluster ID: ae05f165441046f5b00f6482df746c01
I20260925 22:41:24.594357 9922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:41:24.605134 9922 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:41:24.606530 9922 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:41:24.614172 9922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8178a160028d4173af41c7df23c6947b: Generated new TSK 0
I20260925 22:41:24.615005 9922 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.193.keytab.
Entry for principal oryx/127.2.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.193.keytab.
I20260925 22:41:24.646009 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:0
--local_ip_for_outbound_sockets=127.2.74.193
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:38925
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.193.keytab
--principal=oryx/127.2.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:38737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:24.763391 9934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:24.763680 9934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:24.763726 9934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:24.767828 9934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:24.767894 9934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:24.767935 9934 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:41:24.767968 9934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:41:24.773998 9934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:38737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.193.keytab
--principal=oryx/127.2.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.74.254:38925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:24.775125 9934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:24.776120 9934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:24.786303 9942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:24.786286 9939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:24.786343 9934 server_base.cc:1034] running on GCE node
W20260925 22:41:24.786291 9940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:24.787045 9934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:24.787758 9934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:24.788970 9934 hybrid_clock.cc:648] HybridClock initialized: now 1790376084788953 us; error 40 us; skew 500 ppm
Sep 25 22:41:24 dist-test-slave-csnl krb5kdc[9817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376084, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:24.792104 9934 init.cc:377] Logged in from keytab as oryx/127.2.74.193@KRBTEST.COM (short username oryx)
I20260925 22:41:24.793073 9934 webserver.cc:460] Webserver started at http://127.2.74.193:35773/ using document root <none> and password file <none>
I20260925 22:41:24.793686 9934 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:24.793787 9934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:24.794032 9934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:24.796216 9934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-0/data/instance:
uuid: "73b595951f524337992bb66a6d9e9b18"
format_stamp: "Formatted at 2026-09-25 22:41:24 on dist-test-slave-csnl"
server_key: "6c8dbf4e38ff6639c5b4798ef50b8b05"
server_key_iv: "8563bb093c104b8ad3a82d688a22979b"
server_key_version: "encryptionkey@0"
I20260925 22:41:24.796789 9934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance:
uuid: "73b595951f524337992bb66a6d9e9b18"
format_stamp: "Formatted at 2026-09-25 22:41:24 on dist-test-slave-csnl"
server_key: "6c8dbf4e38ff6639c5b4798ef50b8b05"
server_key_iv: "8563bb093c104b8ad3a82d688a22979b"
server_key_version: "encryptionkey@0"
I20260925 22:41:24.800802 9934 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 22:41:24.803551 9949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:24.804903 9934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:41:24.805083 9934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "73b595951f524337992bb66a6d9e9b18"
format_stamp: "Formatted at 2026-09-25 22:41:24 on dist-test-slave-csnl"
server_key: "6c8dbf4e38ff6639c5b4798ef50b8b05"
server_key_iv: "8563bb093c104b8ad3a82d688a22979b"
server_key_version: "encryptionkey@0"
I20260925 22:41:24.805218 9934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:24.828025 9934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:24.831190 9934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:24.831411 9934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:24.833225 9934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:24.833292 9934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:24.833348 9934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:24.833379 9934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:24.846554 9934 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:43217
I20260925 22:41:24.848184 9934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:41:24.848954 10064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:43217 every 8 connection(s)
Sep 25 22:41:24 dist-test-slave-csnl krb5kdc[9817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376084, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM
I20260925 22:41:24.854911 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 9934
I20260925 22:41:24.855094 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance
I20260925 22:41:24.855446 2347 external_mini_cluster.cc:1428] Setting key 46a7956412d54c13ef9e53a4df21a12f
I20260925 22:41:24.856349 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:24.837005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:51715 (local address 127.2.74.254:38925)
0925 22:41:24.837155 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:24.837160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:24.839363 (+ 2203us) server_negotiation.cc:407] Connection header received
0925 22:41:24.841895 (+ 2532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:24.841901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:24.841987 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:24.842084 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:24.844679 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:24.845512 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:24.847394 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:24.847799 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:24.847926 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:24.851822 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:24.851828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:24.851831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:24.851863 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:24.853422 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:24.854017 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:24.854021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:24.854025 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:24.854140 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:24.854640 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:24.854643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:24.854645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:24.855003 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:24.855097 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:24.856035 (+ 938us) server_negotiation.cc:299] Negotiation successful
0925 22:41:24.856176 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:41:24.857472 9958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:24.835528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:38925 (local address 127.2.74.193:51715)
0925 22:41:24.839172 (+ 3644us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:24.839238 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:24.840830 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:24.842464 (+ 1634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:24.842475 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:24.843076 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:24.844453 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:24.844481 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:24.845686 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:24.845692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:24.847215 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:24.847232 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:24.848123 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:24.848129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:24.848351 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:24.849398 (+ 1047us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:24.849429 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:24.851559 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:24.853575 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:24.853582 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:24.853601 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:24.853896 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:24.854258 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:24.854261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:24.854263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:24.854513 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:24.855197 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:24.855204 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:24.855651 (+ 447us) client_negotiation.cc:769] Sending connection context
0925 22:41:24.855963 (+ 312us) client_negotiation.cc:240] Negotiation successful
0925 22:41:24.856328 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3363,"thread_start_us":140,"threads_started":1}
I20260925 22:41:24.858963 10065 heartbeater.cc:347] Connected to a master server at 127.2.74.254:38925
I20260925 22:41:24.859153 10065 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:24.859659 10065 heartbeater.cc:511] Master 127.2.74.254:38925 requested a full tablet report, sending...
I20260925 22:41:24.861451 9861 ts_manager.cc:194] Registered new tserver with Master: 73b595951f524337992bb66a6d9e9b18 (127.2.74.193:43217)
I20260925 22:41:24.863139 9861 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.74.193@KRBTEST.COM'} at 127.2.74.193:51715
WARNING: no policy specified for oryx/127.2.74.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.74.194@KRBTEST.COM" created.
I20260925 22:41:24.871909 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:24.861657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:49335 (local address 127.2.74.254:38925)
0925 22:41:24.861785 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:24.861790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:24.862228 (+ 438us) server_negotiation.cc:407] Connection header received
0925 22:41:24.862331 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:24.862334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:24.862397 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:24.862491 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:24.863062 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:24.863916 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:24.865582 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:24.865954 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:24.866042 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:24.867719 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:24.867725 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:24.867729 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:24.867764 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:24.869664 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:24.870306 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:24.870310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:24.870311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:24.870421 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:24.870755 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:24.870770 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:24.870772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:24.871068 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:24.871187 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:24.871527 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 22:41:24.871669 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 22:41:24.872224 10072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:24.861818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:38925 (local address 127.2.74.193:49335)
0925 22:41:24.862111 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:24.862128 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:24.862243 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:24.862537 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:24.862540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:24.862760 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:24.862885 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:24.862892 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:24.864074 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:24.864081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:24.865400 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:24.865416 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:24.866237 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:24.866241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:24.866355 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:24.867024 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:24.867044 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:24.867546 (+ 502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:24.869815 (+ 2269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:24.869819 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:24.869822 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:24.870156 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:24.870525 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:24.870528 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:24.870530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:24.870668 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:24.871187 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:24.871193 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:24.871327 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 22:41:24.871461 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 22:41:24.871645 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":2,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.194.keytab.
Entry for principal oryx/127.2.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.194.keytab.
I20260925 22:41:24.888293 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:0
--local_ip_for_outbound_sockets=127.2.74.194
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:38925
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.194.keytab
--principal=oryx/127.2.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:38737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:25.004212 10074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:25.004550 10074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:25.004630 10074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:25.008896 10074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:25.009021 10074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:25.009126 10074 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:41:25.009207 10074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:41:25.015209 10074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:38737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.194.keytab
--principal=oryx/127.2.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.74.254:38925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:25.016381 10074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:25.017446 10074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:41:25.027575 10074 server_base.cc:1034] running on GCE node
W20260925 22:41:25.027534 10080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:25.027535 10079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:25.027541 10082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:25.028184 10074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:25.028853 10074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:25.030021 10074 hybrid_clock.cc:648] HybridClock initialized: now 1790376085030001 us; error 28 us; skew 500 ppm
Sep 25 22:41:25 dist-test-slave-csnl krb5kdc[9817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376085, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:25.032853 10074 init.cc:377] Logged in from keytab as oryx/127.2.74.194@KRBTEST.COM (short username oryx)
I20260925 22:41:25.033782 10074 webserver.cc:460] Webserver started at http://127.2.74.194:41097/ using document root <none> and password file <none>
I20260925 22:41:25.034387 10074 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:25.034482 10074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:25.034721 10074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:25.036895 10074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-1/data/instance:
uuid: "82f5b407bb024c43afec7bb9d5e25da5"
format_stamp: "Formatted at 2026-09-25 22:41:25 on dist-test-slave-csnl"
server_key: "e76c5004b9550e97e6f718e2de1bf7e0"
server_key_iv: "e6716ac4924e9c266b6d25cc654bd96f"
server_key_version: "encryptionkey@0"
I20260925 22:41:25.037483 10074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance:
uuid: "82f5b407bb024c43afec7bb9d5e25da5"
format_stamp: "Formatted at 2026-09-25 22:41:25 on dist-test-slave-csnl"
server_key: "e76c5004b9550e97e6f718e2de1bf7e0"
server_key_iv: "e6716ac4924e9c266b6d25cc654bd96f"
server_key_version: "encryptionkey@0"
I20260925 22:41:25.041446 10074 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:41:25.044015 10089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:25.045290 10074 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:41:25.045436 10074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "82f5b407bb024c43afec7bb9d5e25da5"
format_stamp: "Formatted at 2026-09-25 22:41:25 on dist-test-slave-csnl"
server_key: "e76c5004b9550e97e6f718e2de1bf7e0"
server_key_iv: "e6716ac4924e9c266b6d25cc654bd96f"
server_key_version: "encryptionkey@0"
I20260925 22:41:25.045562 10074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:25.065418 10074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:25.068581 10074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:25.068800 10074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:25.070261 10074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:25.070350 10074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:25.070441 10074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:25.070493 10074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:25.080255 10074 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:44861
I20260925 22:41:25.080571 10204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:44861 every 8 connection(s)
Sep 25 22:41:25 dist-test-slave-csnl krb5kdc[9817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376085, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM
I20260925 22:41:25.081243 10074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:41:25.085206 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 10074
I20260925 22:41:25.085345 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance
I20260925 22:41:25.085680 2347 external_mini_cluster.cc:1428] Setting key cd467a2e937f24bdccdd32c8f431ddca
I20260925 22:41:25.085824 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:25.071338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:60719 (local address 127.2.74.254:38925)
0925 22:41:25.071470 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:25.071474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:25.072575 (+ 1101us) server_negotiation.cc:407] Connection header received
0925 22:41:25.073787 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:25.073790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:25.073851 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:25.073925 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:25.075804 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.076716 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.078290 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.078685 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.078820 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:25.081877 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:25.081881 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:25.081884 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:25.081913 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:25.083573 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:25.084198 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:25.084201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:25.084203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:25.084304 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:25.084634 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:25.084639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:25.084642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:25.084888 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:25.085004 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:25.085515 (+ 511us) server_negotiation.cc:299] Negotiation successful
0925 22:41:25.085653 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:41:25.086697 10098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:25.071902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:38925 (local address 127.2.74.194:60719)
0925 22:41:25.072411 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:25.072456 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:25.073507 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:25.074257 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:25.074267 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:25.074765 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:25.075604 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:25.075624 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.076886 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.076891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:25.078148 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:25.078158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.078864 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.078872 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:25.079146 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:25.079886 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:25.079904 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:25.081665 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:25.083693 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:25.083700 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:25.083715 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:25.084027 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:25.084410 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:25.084412 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:25.084413 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:25.084544 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:25.084999 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:25.085006 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:25.085253 (+ 247us) client_negotiation.cc:769] Sending connection context
0925 22:41:25.085460 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 22:41:25.085770 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":144,"threads_started":1}
I20260925 22:41:25.088258 10205 heartbeater.cc:347] Connected to a master server at 127.2.74.254:38925
I20260925 22:41:25.088464 10205 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:25.089129 10205 heartbeater.cc:511] Master 127.2.74.254:38925 requested a full tablet report, sending...
I20260925 22:41:25.090294 9861 ts_manager.cc:194] Registered new tserver with Master: 82f5b407bb024c43afec7bb9d5e25da5 (127.2.74.194:44861)
I20260925 22:41:25.091185 9861 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.74.194@KRBTEST.COM'} at 127.2.74.194:60719
WARNING: no policy specified for oryx/127.2.74.195@KRBTEST.COM; defaulting to no policy
I20260925 22:41:25.099794 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:25.090539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:54127 (local address 127.2.74.254:38925)
0925 22:41:25.090661 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:25.090667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:25.091037 (+ 370us) server_negotiation.cc:407] Connection header received
0925 22:41:25.091137 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:25.091143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:25.091203 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:25.091286 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:25.092031 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.092927 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.094377 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.094600 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.094694 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:25.096186 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:25.096192 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:25.096196 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:25.096230 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:25.097891 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:25.098350 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:25.098353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:25.098354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:25.098435 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:25.098683 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:25.098685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:25.098686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:25.099053 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:25.099138 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:25.099486 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 22:41:25.099657 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:41:25.099792 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:25.090568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:38925 (local address 127.2.74.194:54127)
0925 22:41:25.090895 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:25.090919 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:25.091052 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:25.091422 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:25.091428 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:25.091693 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:25.091865 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:25.091876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.093136 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.093142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:25.094218 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:25.094234 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.094755 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.094759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:25.094867 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:25.095439 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:25.095459 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:25.095956 (+ 497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:25.098004 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:25.098007 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:25.098009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:25.098247 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:25.098518 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:25.098520 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:25.098522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:25.098603 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:25.099131 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:25.099134 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:25.099282 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 22:41:25.099436 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 22:41:25.099591 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.74.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.195.keytab.
Entry for principal oryx/127.2.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.195.keytab.
I20260925 22:41:25.118324 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:0
--local_ip_for_outbound_sockets=127.2.74.195
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:38925
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.195.keytab
--principal=oryx/127.2.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:38737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:25.235432 10214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:25.235750 10214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:25.235800 10214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:25.240279 10214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:25.240370 10214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:25.240412 10214 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:41:25.240483 10214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:41:25.246246 10214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:38737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.195.keytab
--principal=oryx/127.2.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.74.254:38925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:25.247516 10214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:25.248705 10214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:25.258890 10220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:25.258895 10214 server_base.cc:1034] running on GCE node
W20260925 22:41:25.258924 10219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:25.258932 10222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:25.259582 10214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:25.260365 10214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:25.261561 10214 hybrid_clock.cc:648] HybridClock initialized: now 1790376085261539 us; error 35 us; skew 500 ppm
Sep 25 22:41:25 dist-test-slave-csnl krb5kdc[9817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376085, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:25.264511 10214 init.cc:377] Logged in from keytab as oryx/127.2.74.195@KRBTEST.COM (short username oryx)
I20260925 22:41:25.265542 10214 webserver.cc:460] Webserver started at http://127.2.74.195:36997/ using document root <none> and password file <none>
I20260925 22:41:25.266135 10214 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:25.266230 10214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:25.266463 10214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:25.268615 10214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-2/data/instance:
uuid: "a7729df827f14e15aed7a83d71d537dd"
format_stamp: "Formatted at 2026-09-25 22:41:25 on dist-test-slave-csnl"
server_key: "a714ddc324fef5f912e35837526cf448"
server_key_iv: "e5b4a41b5830e6c8969d2809e0b6c636"
server_key_version: "encryptionkey@0"
I20260925 22:41:25.269202 10214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance:
uuid: "a7729df827f14e15aed7a83d71d537dd"
format_stamp: "Formatted at 2026-09-25 22:41:25 on dist-test-slave-csnl"
server_key: "a714ddc324fef5f912e35837526cf448"
server_key_iv: "e5b4a41b5830e6c8969d2809e0b6c636"
server_key_version: "encryptionkey@0"
I20260925 22:41:25.273144 10214 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:41:25.275979 10229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:25.277318 10214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:41:25.277480 10214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "a7729df827f14e15aed7a83d71d537dd"
format_stamp: "Formatted at 2026-09-25 22:41:25 on dist-test-slave-csnl"
server_key: "a714ddc324fef5f912e35837526cf448"
server_key_iv: "e5b4a41b5830e6c8969d2809e0b6c636"
server_key_version: "encryptionkey@0"
I20260925 22:41:25.277619 10214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:25.318235 10214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:25.321669 10214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:25.321900 10214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:25.323571 10214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:25.323652 10214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:25.323727 10214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:25.323769 10214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:41:25 dist-test-slave-csnl krb5kdc[9817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376085, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM
I20260925 22:41:25.335431 10214 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:45449
I20260925 22:41:25.335460 10344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:45449 every 8 connection(s)
I20260925 22:41:25.336691 10214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:41:25.338691 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:25.324321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:58537 (local address 127.2.74.254:38925)
0925 22:41:25.324456 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:25.324460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:25.325632 (+ 1172us) server_negotiation.cc:407] Connection header received
0925 22:41:25.326717 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:25.326722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:25.326788 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:25.326852 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:25.328662 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.329341 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.330759 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.331069 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.331194 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:25.334607 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:25.334611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:25.334614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:25.334647 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:25.336444 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:25.337077 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:25.337081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:25.337083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:25.337192 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:25.337622 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:25.337626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:25.337627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:25.337870 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:25.337958 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:25.338461 (+ 503us) server_negotiation.cc:299] Negotiation successful
0925 22:41:25.338571 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:41:25.339633 10238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:25.324932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:38925 (local address 127.2.74.195:58537)
0925 22:41:25.325452 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:25.325498 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:25.326446 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:25.327294 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:25.327302 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:25.327793 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:25.328489 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:25.328506 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.329475 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.329479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:25.330606 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:25.330617 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.331253 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.331257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:25.331501 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:25.332243 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:25.332265 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:25.334386 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:25.336599 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:25.336604 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:25.336617 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:25.336955 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:25.337314 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:25.337317 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:25.337320 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:25.337503 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:25.337996 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:25.338001 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:25.338248 (+ 247us) client_negotiation.cc:769] Sending connection context
0925 22:41:25.338408 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 22:41:25.338670 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":143,"threads_started":1}
I20260925 22:41:25.340822 10345 heartbeater.cc:347] Connected to a master server at 127.2.74.254:38925
I20260925 22:41:25.340970 10345 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:25.341563 10345 heartbeater.cc:511] Master 127.2.74.254:38925 requested a full tablet report, sending...
I20260925 22:41:25.342744 9861 ts_manager.cc:194] Registered new tserver with Master: a7729df827f14e15aed7a83d71d537dd (127.2.74.195:45449)
I20260925 22:41:25.343572 9861 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.74.195@KRBTEST.COM'} at 127.2.74.195:58537
I20260925 22:41:25.346184 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 10214
I20260925 22:41:25.346340 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance
I20260925 22:41:25.346635 2347 external_mini_cluster.cc:1428] Setting key 8d3ef7e90ed4dfd338c9721d7846de62
I20260925 22:41:25.348766 2347 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:41:25.351262 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:25.343289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:34837 (local address 127.2.74.254:38925)
0925 22:41:25.343409 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:25.343412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:25.344067 (+ 655us) server_negotiation.cc:407] Connection header received
0925 22:41:25.344206 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:25.344210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:25.344257 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:25.344341 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:25.344769 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.345491 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.346292 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.346567 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.346676 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:25.347878 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:25.347881 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:25.347884 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:25.347908 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:25.349406 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:25.349901 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:25.349904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:25.349906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:25.349985 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:25.350262 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:25.350267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:25.350269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:25.350509 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:25.350626 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:25.351034 (+ 408us) server_negotiation.cc:299] Negotiation successful
0925 22:41:25.351145 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:41:25.351353 10351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:25.343223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:38925 (local address 127.2.74.195:34837)
0925 22:41:25.343977 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:25.343992 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:25.344103 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:25.344363 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:25.344365 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:25.344556 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:25.344659 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:25.344664 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.345614 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.345617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:25.346167 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:25.346174 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.346681 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.346685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:25.346776 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:25.347351 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:25.347362 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:25.347750 (+ 388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:25.349534 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:25.349538 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:25.349540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:25.349755 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:25.350084 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:25.350087 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:25.350089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:25.350172 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:25.350677 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:25.350681 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:25.350792 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 22:41:25.350908 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 22:41:25.351112 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":106,"threads_started":1}
Sep 25 22:41:25 dist-test-slave-csnl krb5kdc[9817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376085, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:41:25 dist-test-slave-csnl krb5kdc[9817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:25 dist-test-slave-csnl krb5kdc[9817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 22:41:25.373970 9911 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:25.367290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42828 (local address 127.2.74.254:38925)
0925 22:41:25.367427 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:25.367430 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:25.367443 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:41:25.367481 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:25.367483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:25.367521 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:25.367620 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:25.368355 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.368884 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.369768 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.369990 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.370119 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:25.373784 (+ 3665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42828: BlockingRecv error: recv got EOF from 127.0.0.1:42828 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 22:41:25 dist-test-slave-csnl krb5kdc[9817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376085, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM
I20260925 22:41:25.384774 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:25.375693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42836 (local address 127.2.74.254:38925)
0925 22:41:25.375830 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:25.375834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:25.375957 (+ 123us) server_negotiation.cc:407] Connection header received
0925 22:41:25.376133 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:25.376136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:25.376191 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:25.376280 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:25.376918 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.377440 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.378265 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.378499 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.378618 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:25.381204 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:25.381209 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:25.381211 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:25.381250 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:25.382690 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:25.383314 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:25.383317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:25.383318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:25.383410 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:25.383725 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:25.383728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:25.383729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:25.383946 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:25.384100 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:25.384401 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 22:41:25.384590 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 22:41:25.388159 9861 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:41:25.391469 9861 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:41:25.407126 10367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:25.401096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:45449 (local address 127.0.0.1:50612)
0925 22:41:25.402304 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:25.402317 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:25.402394 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:25.402935 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:25.402938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:25.402985 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:25.403097 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:25.403106 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.404507 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.404510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:25.405640 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:25.405648 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.406491 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.406494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:25.406627 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:25.406674 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 22:41:25.406798 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 22:41:25.406934 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1149,"mutex_wait_us":34,"thread_start_us":94,"threads_started":1}
I20260925 22:41:25.407136 10371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:25.401500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50612 (local address 127.2.74.195:45449)
0925 22:41:25.402254 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:25.402266 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:25.402490 (+ 224us) server_negotiation.cc:407] Connection header received
0925 22:41:25.402573 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:25.402577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:25.402797 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:25.402981 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:25.403221 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.404355 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.405744 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.406355 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.406490 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:25.406693 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:25.406830 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 22:41:25.406934 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":112,"threads_started":1}
I20260925 22:41:25.407303 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:25.400813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45530 (local address 127.2.74.193:43217)
0925 22:41:25.401258 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:25.401266 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:25.401677 (+ 411us) server_negotiation.cc:407] Connection header received
0925 22:41:25.401936 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:25.401942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:25.402176 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:25.402346 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:25.402675 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.403733 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.405510 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.406370 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.406457 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:25.406625 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:25.407041 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 22:41:25.407149 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":91,"threads_started":1}
I20260925 22:41:25.407615 10369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:25.400838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:44861 (local address 127.0.0.1:38928)
0925 22:41:25.401870 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:25.401884 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:25.401987 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:25.403253 (+ 1266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:25.403256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:25.403272 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:25.403352 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:25.403357 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.404669 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.404675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:25.405764 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:25.405774 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.407088 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.407101 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:25.407194 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:25.407217 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:41:25.407378 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 22:41:25.407467 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"mutex_wait_us":4,"thread_start_us":99,"threads_started":1}
I20260925 22:41:25.407895 10370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:25.400669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:43217 (local address 127.0.0.1:45530)
0925 22:41:25.401519 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:25.401566 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:25.401735 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:25.402350 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:25.402356 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:25.402377 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:25.402534 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:25.402541 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.403841 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.403847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:25.405132 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:25.405382 (+ 250us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.406732 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.406735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:25.406833 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:25.406859 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 22:41:25.407518 (+ 659us) client_negotiation.cc:240] Negotiation successful
0925 22:41:25.407605 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":45,"thread_start_us":103,"threads_started":1}
I20260925 22:41:25.408097 10372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:25.401500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38928 (local address 127.2.74.194:44861)
0925 22:41:25.402114 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:25.402127 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:25.402149 (+ 22us) server_negotiation.cc:407] Connection header received
0925 22:41:25.402729 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:25.402736 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:25.402929 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:25.403116 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:25.403477 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.404563 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.405925 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.406734 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.406834 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:25.407029 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:25.407846 (+ 817us) server_negotiation.cc:299] Negotiation successful
0925 22:41:25.407956 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":133,"threads_started":1}
I20260925 22:41:25.410778 9990 tablet_service.cc:1467] Processing CreateTablet for tablet 58d0bf33449444548648d93c6618db7e (DEFAULT_TABLE table=test-table [id=122a6b3fb9a846eab2911b5625445dd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:25.410804 10279 tablet_service.cc:1467] Processing CreateTablet for tablet 58d0bf33449444548648d93c6618db7e (DEFAULT_TABLE table=test-table [id=122a6b3fb9a846eab2911b5625445dd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:25.411300 10139 tablet_service.cc:1467] Processing CreateTablet for tablet 58d0bf33449444548648d93c6618db7e (DEFAULT_TABLE table=test-table [id=122a6b3fb9a846eab2911b5625445dd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:25.411935 9990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58d0bf33449444548648d93c6618db7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:25.411940 10279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58d0bf33449444548648d93c6618db7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:25.412261 10139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58d0bf33449444548648d93c6618db7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:25.418205 10373 tablet_bootstrap.cc:492] T 58d0bf33449444548648d93c6618db7e P 73b595951f524337992bb66a6d9e9b18: Bootstrap starting.
I20260925 22:41:25.419771 10375 tablet_bootstrap.cc:492] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5: Bootstrap starting.
I20260925 22:41:25.420212 10374 tablet_bootstrap.cc:492] T 58d0bf33449444548648d93c6618db7e P a7729df827f14e15aed7a83d71d537dd: Bootstrap starting.
I20260925 22:41:25.421931 10373 tablet_bootstrap.cc:654] T 58d0bf33449444548648d93c6618db7e P 73b595951f524337992bb66a6d9e9b18: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:25.422576 10375 tablet_bootstrap.cc:654] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:25.422878 10373 log.cc:824] T 58d0bf33449444548648d93c6618db7e P 73b595951f524337992bb66a6d9e9b18: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:25.423452 10375 log.cc:824] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:25.425202 10373 tablet_bootstrap.cc:492] T 58d0bf33449444548648d93c6618db7e P 73b595951f524337992bb66a6d9e9b18: No bootstrap required, opened a new log
I20260925 22:41:25.425297 10374 tablet_bootstrap.cc:654] T 58d0bf33449444548648d93c6618db7e P a7729df827f14e15aed7a83d71d537dd: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:25.425443 10373 ts_tablet_manager.cc:1397] T 58d0bf33449444548648d93c6618db7e P 73b595951f524337992bb66a6d9e9b18: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 22:41:25.425885 10375 tablet_bootstrap.cc:492] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5: No bootstrap required, opened a new log
I20260925 22:41:25.426093 10375 ts_tablet_manager.cc:1397] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:41:25.426154 10374 log.cc:824] T 58d0bf33449444548648d93c6618db7e P a7729df827f14e15aed7a83d71d537dd: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:25.428381 10374 tablet_bootstrap.cc:492] T 58d0bf33449444548648d93c6618db7e P a7729df827f14e15aed7a83d71d537dd: No bootstrap required, opened a new log
I20260925 22:41:25.428604 10374 ts_tablet_manager.cc:1397] T 58d0bf33449444548648d93c6618db7e P a7729df827f14e15aed7a83d71d537dd: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260925 22:41:25.429353 10373 raft_consensus.cc:348] T 58d0bf33449444548648d93c6618db7e P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:25.429634 10373 raft_consensus.cc:374] T 58d0bf33449444548648d93c6618db7e P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:25.429685 10373 raft_consensus.cc:729] T 58d0bf33449444548648d93c6618db7e P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73b595951f524337992bb66a6d9e9b18, State: Initialized, Role: FOLLOWER
I20260925 22:41:25.429963 10375 raft_consensus.cc:348] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:25.430225 10375 raft_consensus.cc:374] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:25.430272 10375 raft_consensus.cc:729] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f5b407bb024c43afec7bb9d5e25da5, State: Initialized, Role: FOLLOWER
I20260925 22:41:25.430186 10373 consensus_queue.cc:260] T 58d0bf33449444548648d93c6618db7e P 73b595951f524337992bb66a6d9e9b18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:25.430756 10375 consensus_queue.cc:260] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:25.431331 10373 ts_tablet_manager.cc:1428] T 58d0bf33449444548648d93c6618db7e P 73b595951f524337992bb66a6d9e9b18: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260925 22:41:25.431577 10205 heartbeater.cc:503] Master 127.2.74.254:38925 was elected leader, sending a full tablet report...
I20260925 22:41:25.431666 10065 heartbeater.cc:503] Master 127.2.74.254:38925 was elected leader, sending a full tablet report...
I20260925 22:41:25.431814 10375 ts_tablet_manager.cc:1428] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260925 22:41:25.432657 10374 raft_consensus.cc:348] T 58d0bf33449444548648d93c6618db7e P a7729df827f14e15aed7a83d71d537dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:25.432987 10374 raft_consensus.cc:374] T 58d0bf33449444548648d93c6618db7e P a7729df827f14e15aed7a83d71d537dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:25.433054 10374 raft_consensus.cc:729] T 58d0bf33449444548648d93c6618db7e P a7729df827f14e15aed7a83d71d537dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7729df827f14e15aed7a83d71d537dd, State: Initialized, Role: FOLLOWER
I20260925 22:41:25.433537 10374 consensus_queue.cc:260] T 58d0bf33449444548648d93c6618db7e P a7729df827f14e15aed7a83d71d537dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:25.434285 10345 heartbeater.cc:503] Master 127.2.74.254:38925 was elected leader, sending a full tablet report...
I20260925 22:41:25.434700 10374 ts_tablet_manager.cc:1428] T 58d0bf33449444548648d93c6618db7e P a7729df827f14e15aed7a83d71d537dd: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260925 22:41:25.582535 10206 tablet.cc:2369] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:41:25.587747 10346 tablet.cc:2369] T 58d0bf33449444548648d93c6618db7e P a7729df827f14e15aed7a83d71d537dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:41:25.599742 10066 tablet.cc:2369] T 58d0bf33449444548648d93c6618db7e P 73b595951f524337992bb66a6d9e9b18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:41:25.611631 10380 raft_consensus.cc:482] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:41:25.611831 10380 raft_consensus.cc:504] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:25.613049 10380 leader_election.cc:288] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73b595951f524337992bb66a6d9e9b18 (127.2.74.193:43217), a7729df827f14e15aed7a83d71d537dd (127.2.74.195:45449)
I20260925 22:41:25.616888 10371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:25.613470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:45505 (local address 127.2.74.195:45449)
0925 22:41:25.613661 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:25.613667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:25.613709 (+ 42us) server_negotiation.cc:407] Connection header received
0925 22:41:25.613866 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:25.613868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:25.613917 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:25.613999 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:25.614265 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.614768 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.615929 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.616331 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.616457 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:25.616511 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:25.616689 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 22:41:25.616754 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 22:41:25.616894 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:25.613262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:54479 (local address 127.2.74.193:43217)
0925 22:41:25.613404 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:25.613409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:25.613654 (+ 245us) server_negotiation.cc:407] Connection header received
0925 22:41:25.613828 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:25.613831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:25.613875 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:25.613975 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:25.614256 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.614781 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.615912 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.616330 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.616463 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:25.616513 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:25.616704 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 22:41:25.616772 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:41:25.616905 10382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:25.613202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:43217 (local address 127.2.74.194:54479)
0925 22:41:25.613542 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:25.613565 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:25.613701 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:25.613988 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:25.613991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:25.614007 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:25.614145 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:25.614150 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.614903 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.614906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:25.615783 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:25.615792 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.616463 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.616466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:25.616583 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:25.616600 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 22:41:25.616688 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 22:41:25.616759 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":9,"spinlock_wait_cycles":22784,"thread_start_us":77,"threads_started":1}
I20260925 22:41:25.616915 10383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:25.613289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:45449 (local address 127.2.74.194:45505)
0925 22:41:25.613639 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:25.613653 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:25.613763 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:25.614018 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:25.614021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:25.614058 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:25.614162 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:25.614167 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.614876 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.614881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:25.615787 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:25.615796 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.616451 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.616454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:25.616556 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:25.616578 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:41:25.616674 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 22:41:25.616752 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":37,"spinlock_wait_cycles":896,"thread_start_us":59,"threads_started":1}
I20260925 22:41:25.617558 10006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d0bf33449444548648d93c6618db7e" candidate_uuid: "82f5b407bb024c43afec7bb9d5e25da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73b595951f524337992bb66a6d9e9b18" is_pre_election: true
I20260925 22:41:25.617673 10299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d0bf33449444548648d93c6618db7e" candidate_uuid: "82f5b407bb024c43afec7bb9d5e25da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7729df827f14e15aed7a83d71d537dd" is_pre_election: true
I20260925 22:41:25.617920 10006 raft_consensus.cc:2454] T 58d0bf33449444548648d93c6618db7e P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82f5b407bb024c43afec7bb9d5e25da5 in term 0.
I20260925 22:41:25.617967 10299 raft_consensus.cc:2454] T 58d0bf33449444548648d93c6618db7e P a7729df827f14e15aed7a83d71d537dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82f5b407bb024c43afec7bb9d5e25da5 in term 0.
I20260925 22:41:25.618422 10091 leader_election.cc:302] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 73b595951f524337992bb66a6d9e9b18, 82f5b407bb024c43afec7bb9d5e25da5, a7729df827f14e15aed7a83d71d537dd; no voters:
I20260925 22:41:25.618711 10380 raft_consensus.cc:2781] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:41:25.618795 10380 raft_consensus.cc:482] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:41:25.618837 10380 raft_consensus.cc:3037] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:25.619907 10380 raft_consensus.cc:504] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:25.620350 10380 leader_election.cc:288] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5 [CANDIDATE]: Term 1 election: Requested vote from peers 73b595951f524337992bb66a6d9e9b18 (127.2.74.193:43217), a7729df827f14e15aed7a83d71d537dd (127.2.74.195:45449)
I20260925 22:41:25.620671 10006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d0bf33449444548648d93c6618db7e" candidate_uuid: "82f5b407bb024c43afec7bb9d5e25da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73b595951f524337992bb66a6d9e9b18"
I20260925 22:41:25.620796 10006 raft_consensus.cc:3037] T 58d0bf33449444548648d93c6618db7e P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:25.620920 10299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d0bf33449444548648d93c6618db7e" candidate_uuid: "82f5b407bb024c43afec7bb9d5e25da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7729df827f14e15aed7a83d71d537dd"
I20260925 22:41:25.621038 10299 raft_consensus.cc:3037] T 58d0bf33449444548648d93c6618db7e P a7729df827f14e15aed7a83d71d537dd [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:25.622263 10006 raft_consensus.cc:2454] T 58d0bf33449444548648d93c6618db7e P 73b595951f524337992bb66a6d9e9b18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82f5b407bb024c43afec7bb9d5e25da5 in term 1.
I20260925 22:41:25.622642 10091 leader_election.cc:302] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73b595951f524337992bb66a6d9e9b18, 82f5b407bb024c43afec7bb9d5e25da5; no voters:
I20260925 22:41:25.622790 10299 raft_consensus.cc:2454] T 58d0bf33449444548648d93c6618db7e P a7729df827f14e15aed7a83d71d537dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82f5b407bb024c43afec7bb9d5e25da5 in term 1.
I20260925 22:41:25.622845 10380 raft_consensus.cc:2781] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:41:25.623088 10380 raft_consensus.cc:686] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5 [term 1 LEADER]: Becoming Leader. State: Replica: 82f5b407bb024c43afec7bb9d5e25da5, State: Running, Role: LEADER
I20260925 22:41:25.623472 10380 consensus_queue.cc:237] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:25.626832 9861 catalog_manager.cc:5495] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82f5b407bb024c43afec7bb9d5e25da5 (127.2.74.194). New cstate: current_term: 1 leader_uuid: "82f5b407bb024c43afec7bb9d5e25da5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } health_report { overall_health: UNKNOWN } } }
I20260925 22:41:25.651691 10372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:25.647413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38934 (local address 127.2.74.194:44861)
0925 22:41:25.647534 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:25.647538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:25.647550 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:41:25.647686 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:25.647689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:25.647747 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:25.647840 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:41:25.648047 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.648605 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.650011 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.650282 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.650394 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:25.650704 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:41:25.651267 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:41:25.651398 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:25.651460 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 22:41:25.651528 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:41:25.657763 10299 raft_consensus.cc:1261] T 58d0bf33449444548648d93c6618db7e P a7729df827f14e15aed7a83d71d537dd [term 1 FOLLOWER]: Refusing update from remote peer 82f5b407bb024c43afec7bb9d5e25da5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:41:25.657763 10006 raft_consensus.cc:1261] T 58d0bf33449444548648d93c6618db7e P 73b595951f524337992bb66a6d9e9b18 [term 1 FOLLOWER]: Refusing update from remote peer 82f5b407bb024c43afec7bb9d5e25da5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:41:25.658581 10380 consensus_queue.cc:1035] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:41:25.658802 10384 consensus_queue.cc:1035] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:41:25.667423 10387 mvcc.cc:205] Tried to move back new op lower bound from 7333380446846509056 to 7333380446716153856. Current Snapshot: MvccSnapshot[applied={T|T < 7333380446846509056}]
I20260925 22:41:25.668047 10389 mvcc.cc:205] Tried to move back new op lower bound from 7333380446846509056 to 7333380446716153856. Current Snapshot: MvccSnapshot[applied={T|T < 7333380446846509056}]
I20260925 22:41:25.670648 10388 mvcc.cc:205] Tried to move back new op lower bound from 7333380446846509056 to 7333380446716153856. Current Snapshot: MvccSnapshot[applied={T|T < 7333380446846509056}]
I20260925 22:41:25.680137 9861 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:41:25.680347 9861 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:41:25.683270 9861 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8178a160028d4173af41c7df23c6947b: Sending DeleteTablet for 3 replicas of tablet 58d0bf33449444548648d93c6618db7e
I20260925 22:41:25.684288 9990 tablet_service.cc:1514] Processing DeleteTablet for tablet 58d0bf33449444548648d93c6618db7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:25 UTC) from {username='oryx', principal='oryx/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:45530
I20260925 22:41:25.684285 10139 tablet_service.cc:1514] Processing DeleteTablet for tablet 58d0bf33449444548648d93c6618db7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:25 UTC) from {username='oryx', principal='oryx/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:38928
I20260925 22:41:25.684918 9861 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 22:41:25.685088 10279 tablet_service.cc:1514] Processing DeleteTablet for tablet 58d0bf33449444548648d93c6618db7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:25 UTC) from {username='oryx', principal='oryx/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:50612
W20260925 22:41:25.685375 9861 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:41:25.685477 10398 tablet_replica.cc:330] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5: stopping tablet replica
I20260925 22:41:25.685880 10398 raft_consensus.cc:2229] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:41:25.686353 10398 raft_consensus.cc:2258] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:41:25.687155 10399 tablet_replica.cc:330] T 58d0bf33449444548648d93c6618db7e P a7729df827f14e15aed7a83d71d537dd: stopping tablet replica
I20260925 22:41:25.687490 10399 raft_consensus.cc:2229] T 58d0bf33449444548648d93c6618db7e P a7729df827f14e15aed7a83d71d537dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:41:25.687867 10399 raft_consensus.cc:2258] T 58d0bf33449444548648d93c6618db7e P a7729df827f14e15aed7a83d71d537dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:41:25.689522 10399 ts_tablet_manager.cc:1905] T 58d0bf33449444548648d93c6618db7e P a7729df827f14e15aed7a83d71d537dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:41:25.689761 10398 ts_tablet_manager.cc:1905] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:41:25.691478 9990 tablet_service.cc:1467] Processing CreateTablet for tablet ccc99a712aac4098afa8121e6aa55ca9 (DEFAULT_TABLE table=test-table [id=758206a8f4a049c1890251391d55018d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:25.691773 9990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccc99a712aac4098afa8121e6aa55ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:25.692567 10399 ts_tablet_manager.cc:1918] T 58d0bf33449444548648d93c6618db7e P a7729df827f14e15aed7a83d71d537dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:41:25.692682 10399 log.cc:1196] T 58d0bf33449444548648d93c6618db7e P a7729df827f14e15aed7a83d71d537dd: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-2/wal/wals/58d0bf33449444548648d93c6618db7e
I20260925 22:41:25.692802 10279 tablet_service.cc:1467] Processing CreateTablet for tablet ccc99a712aac4098afa8121e6aa55ca9 (DEFAULT_TABLE table=test-table [id=758206a8f4a049c1890251391d55018d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:25.693049 10399 ts_tablet_manager.cc:1939] T 58d0bf33449444548648d93c6618db7e P a7729df827f14e15aed7a83d71d537dd: Deleting consensus metadata
I20260925 22:41:25.693068 10279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccc99a712aac4098afa8121e6aa55ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:25.693099 10139 tablet_service.cc:1467] Processing CreateTablet for tablet ccc99a712aac4098afa8121e6aa55ca9 (DEFAULT_TABLE table=test-table [id=758206a8f4a049c1890251391d55018d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:25.693360 10139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccc99a712aac4098afa8121e6aa55ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:25.694543 9849 catalog_manager.cc:4841] TS a7729df827f14e15aed7a83d71d537dd (127.2.74.195:45449): tablet 58d0bf33449444548648d93c6618db7e (table test-table [id=122a6b3fb9a846eab2911b5625445dd3]) successfully deleted
I20260925 22:41:25.695497 10374 tablet_bootstrap.cc:492] T ccc99a712aac4098afa8121e6aa55ca9 P a7729df827f14e15aed7a83d71d537dd: Bootstrap starting.
I20260925 22:41:25.695978 10375 tablet_bootstrap.cc:492] T ccc99a712aac4098afa8121e6aa55ca9 P 82f5b407bb024c43afec7bb9d5e25da5: Bootstrap starting.
I20260925 22:41:25.696624 10374 tablet_bootstrap.cc:654] T ccc99a712aac4098afa8121e6aa55ca9 P a7729df827f14e15aed7a83d71d537dd: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:25.697005 10375 tablet_bootstrap.cc:654] T ccc99a712aac4098afa8121e6aa55ca9 P 82f5b407bb024c43afec7bb9d5e25da5: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:25.697943 10374 tablet_bootstrap.cc:492] T ccc99a712aac4098afa8121e6aa55ca9 P a7729df827f14e15aed7a83d71d537dd: No bootstrap required, opened a new log
I20260925 22:41:25.698035 10374 ts_tablet_manager.cc:1397] T ccc99a712aac4098afa8121e6aa55ca9 P a7729df827f14e15aed7a83d71d537dd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:41:25.698496 10375 tablet_bootstrap.cc:492] T ccc99a712aac4098afa8121e6aa55ca9 P 82f5b407bb024c43afec7bb9d5e25da5: No bootstrap required, opened a new log
I20260925 22:41:25.698576 10375 ts_tablet_manager.cc:1397] T ccc99a712aac4098afa8121e6aa55ca9 P 82f5b407bb024c43afec7bb9d5e25da5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260925 22:41:25.698519 10374 raft_consensus.cc:348] T ccc99a712aac4098afa8121e6aa55ca9 P a7729df827f14e15aed7a83d71d537dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:25.698637 10374 raft_consensus.cc:374] T ccc99a712aac4098afa8121e6aa55ca9 P a7729df827f14e15aed7a83d71d537dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:25.698669 10374 raft_consensus.cc:729] T ccc99a712aac4098afa8121e6aa55ca9 P a7729df827f14e15aed7a83d71d537dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7729df827f14e15aed7a83d71d537dd, State: Initialized, Role: FOLLOWER
I20260925 22:41:25.699045 10375 raft_consensus.cc:348] T ccc99a712aac4098afa8121e6aa55ca9 P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:25.699189 10375 raft_consensus.cc:374] T ccc99a712aac4098afa8121e6aa55ca9 P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:25.699222 10375 raft_consensus.cc:729] T ccc99a712aac4098afa8121e6aa55ca9 P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f5b407bb024c43afec7bb9d5e25da5, State: Initialized, Role: FOLLOWER
I20260925 22:41:25.699268 10398 ts_tablet_manager.cc:1918] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:41:25.699344 10375 consensus_queue.cc:260] T ccc99a712aac4098afa8121e6aa55ca9 P 82f5b407bb024c43afec7bb9d5e25da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:25.699617 10375 ts_tablet_manager.cc:1428] T ccc99a712aac4098afa8121e6aa55ca9 P 82f5b407bb024c43afec7bb9d5e25da5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:41:25.700172 10374 consensus_queue.cc:260] T ccc99a712aac4098afa8121e6aa55ca9 P a7729df827f14e15aed7a83d71d537dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:25.700546 10374 ts_tablet_manager.cc:1428] T ccc99a712aac4098afa8121e6aa55ca9 P a7729df827f14e15aed7a83d71d537dd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 22:41:25.701579 10373 tablet_bootstrap.cc:492] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18: Bootstrap starting.
I20260925 22:41:25.699352 10398 log.cc:1196] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-1/wal/wals/58d0bf33449444548648d93c6618db7e
I20260925 22:41:25.702666 10373 tablet_bootstrap.cc:654] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:25.702716 10398 ts_tablet_manager.cc:1939] T 58d0bf33449444548648d93c6618db7e P 82f5b407bb024c43afec7bb9d5e25da5: Deleting consensus metadata
I20260925 22:41:25.704007 10373 tablet_bootstrap.cc:492] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18: No bootstrap required, opened a new log
I20260925 22:41:25.704118 10373 ts_tablet_manager.cc:1397] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:41:25.704100 9848 catalog_manager.cc:4841] TS 82f5b407bb024c43afec7bb9d5e25da5 (127.2.74.194:44861): tablet 58d0bf33449444548648d93c6618db7e (table test-table [id=122a6b3fb9a846eab2911b5625445dd3]) successfully deleted
I20260925 22:41:25.704360 10397 tablet_replica.cc:330] T 58d0bf33449444548648d93c6618db7e P 73b595951f524337992bb66a6d9e9b18: stopping tablet replica
I20260925 22:41:25.704589 10373 raft_consensus.cc:348] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:25.704710 10373 raft_consensus.cc:374] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:25.704731 10397 raft_consensus.cc:2229] T 58d0bf33449444548648d93c6618db7e P 73b595951f524337992bb66a6d9e9b18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:41:25.704749 10373 raft_consensus.cc:729] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73b595951f524337992bb66a6d9e9b18, State: Initialized, Role: FOLLOWER
I20260925 22:41:25.704882 10373 consensus_queue.cc:260] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:25.705173 10397 raft_consensus.cc:2258] T 58d0bf33449444548648d93c6618db7e P 73b595951f524337992bb66a6d9e9b18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:41:25.705236 10373 ts_tablet_manager.cc:1428] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:41:25.706645 10397 ts_tablet_manager.cc:1905] T 58d0bf33449444548648d93c6618db7e P 73b595951f524337992bb66a6d9e9b18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:41:25.709630 10397 ts_tablet_manager.cc:1918] T 58d0bf33449444548648d93c6618db7e P 73b595951f524337992bb66a6d9e9b18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:41:25.709712 10397 log.cc:1196] T 58d0bf33449444548648d93c6618db7e P 73b595951f524337992bb66a6d9e9b18: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-0/wal/wals/58d0bf33449444548648d93c6618db7e
I20260925 22:41:25.710060 10397 ts_tablet_manager.cc:1939] T 58d0bf33449444548648d93c6618db7e P 73b595951f524337992bb66a6d9e9b18: Deleting consensus metadata
I20260925 22:41:25.711256 9847 catalog_manager.cc:4841] TS 73b595951f524337992bb66a6d9e9b18 (127.2.74.193:43217): tablet 58d0bf33449444548648d93c6618db7e (table test-table [id=122a6b3fb9a846eab2911b5625445dd3]) successfully deleted
I20260925 22:41:25.719591 10379 raft_consensus.cc:482] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:41:25.719744 10379 raft_consensus.cc:504] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:25.720988 10379 leader_election.cc:288] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82f5b407bb024c43afec7bb9d5e25da5 (127.2.74.194:44861), a7729df827f14e15aed7a83d71d537dd (127.2.74.195:45449)
I20260925 22:41:25.725943 10371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:25.721533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:39163 (local address 127.2.74.195:45449)
0925 22:41:25.721680 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:25.721685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:25.722031 (+ 346us) server_negotiation.cc:407] Connection header received
0925 22:41:25.722222 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:25.722225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:25.722270 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:25.722340 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:25.722691 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.723327 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.724378 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.724996 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.725123 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:25.725198 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:25.725731 (+ 533us) server_negotiation.cc:299] Negotiation successful
0925 22:41:25.725816 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:41:25.725945 10372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:25.721691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:47289 (local address 127.2.74.194:44861)
0925 22:41:25.721796 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:25.721801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:25.721814 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:41:25.721865 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:25.721868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:25.721913 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:25.721994 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:25.722324 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.723000 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.724508 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.725119 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.725228 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:25.725273 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:25.725723 (+ 450us) server_negotiation.cc:299] Negotiation successful
0925 22:41:25.725807 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 22:41:25.726476 10404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:25.721232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:45449 (local address 127.2.74.193:39163)
0925 22:41:25.721938 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:25.721949 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:25.722108 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:25.722438 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:25.722441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:25.722457 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:25.722571 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:25.722576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.723433 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.723440 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:25.724248 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:25.724257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.725130 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.725134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:25.725579 (+ 445us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:25.725606 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 22:41:25.726156 (+ 550us) client_negotiation.cc:240] Negotiation successful
0925 22:41:25.726306 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":73,"spinlock_wait_cycles":1152,"thread_start_us":92,"threads_started":1}
I20260925 22:41:25.726714 10403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:25.721174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:44861 (local address 127.2.74.193:47289)
0925 22:41:25.721411 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:25.721433 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:25.721535 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:25.722028 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:25.722031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:25.722053 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:25.722219 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:25.722225 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.723113 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.723118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:25.724374 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:25.724385 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.725345 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.725348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:25.725573 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:25.725601 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 22:41:25.726400 (+ 799us) client_negotiation.cc:240] Negotiation successful
0925 22:41:25.726464 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":31,"spinlock_wait_cycles":13696,"thread_start_us":74,"threads_started":1}
I20260925 22:41:25.726938 10299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc99a712aac4098afa8121e6aa55ca9" candidate_uuid: "73b595951f524337992bb66a6d9e9b18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7729df827f14e15aed7a83d71d537dd" is_pre_election: true
I20260925 22:41:25.727142 10299 raft_consensus.cc:2454] T ccc99a712aac4098afa8121e6aa55ca9 P a7729df827f14e15aed7a83d71d537dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73b595951f524337992bb66a6d9e9b18 in term 0.
I20260925 22:41:25.727257 10159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc99a712aac4098afa8121e6aa55ca9" candidate_uuid: "73b595951f524337992bb66a6d9e9b18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f5b407bb024c43afec7bb9d5e25da5" is_pre_election: true
I20260925 22:41:25.727445 10159 raft_consensus.cc:2454] T ccc99a712aac4098afa8121e6aa55ca9 P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73b595951f524337992bb66a6d9e9b18 in term 0.
I20260925 22:41:25.727509 9953 leader_election.cc:302] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73b595951f524337992bb66a6d9e9b18, a7729df827f14e15aed7a83d71d537dd; no voters:
I20260925 22:41:25.727783 10379 raft_consensus.cc:2781] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:41:25.727840 10379 raft_consensus.cc:482] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:41:25.727883 10379 raft_consensus.cc:3037] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:25.729139 10379 raft_consensus.cc:504] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:25.729542 10379 leader_election.cc:288] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18 [CANDIDATE]: Term 1 election: Requested vote from peers 82f5b407bb024c43afec7bb9d5e25da5 (127.2.74.194:44861), a7729df827f14e15aed7a83d71d537dd (127.2.74.195:45449)
I20260925 22:41:25.729911 10299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc99a712aac4098afa8121e6aa55ca9" candidate_uuid: "73b595951f524337992bb66a6d9e9b18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7729df827f14e15aed7a83d71d537dd"
I20260925 22:41:25.729955 10159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc99a712aac4098afa8121e6aa55ca9" candidate_uuid: "73b595951f524337992bb66a6d9e9b18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f5b407bb024c43afec7bb9d5e25da5"
I20260925 22:41:25.730015 10299 raft_consensus.cc:3037] T ccc99a712aac4098afa8121e6aa55ca9 P a7729df827f14e15aed7a83d71d537dd [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:25.730068 10159 raft_consensus.cc:3037] T ccc99a712aac4098afa8121e6aa55ca9 P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:25.731361 10299 raft_consensus.cc:2454] T ccc99a712aac4098afa8121e6aa55ca9 P a7729df827f14e15aed7a83d71d537dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73b595951f524337992bb66a6d9e9b18 in term 1.
I20260925 22:41:25.731765 9953 leader_election.cc:302] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73b595951f524337992bb66a6d9e9b18, a7729df827f14e15aed7a83d71d537dd; no voters:
I20260925 22:41:25.731980 10379 raft_consensus.cc:2781] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:41:25.732215 10379 raft_consensus.cc:686] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18 [term 1 LEADER]: Becoming Leader. State: Replica: 73b595951f524337992bb66a6d9e9b18, State: Running, Role: LEADER
I20260925 22:41:25.732355 10159 raft_consensus.cc:2454] T ccc99a712aac4098afa8121e6aa55ca9 P 82f5b407bb024c43afec7bb9d5e25da5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73b595951f524337992bb66a6d9e9b18 in term 1.
I20260925 22:41:25.732508 10379 consensus_queue.cc:237] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:25.735119 9860 catalog_manager.cc:5495] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73b595951f524337992bb66a6d9e9b18 (127.2.74.193). New cstate: current_term: 1 leader_uuid: "73b595951f524337992bb66a6d9e9b18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } health_report { overall_health: UNKNOWN } } }
Sep 25 22:41:25 dist-test-slave-csnl krb5kdc[9817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376084, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.254@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM
I20260925 22:41:25.786654 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:25.777077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42840 (local address 127.2.74.254:38925)
0925 22:41:25.777217 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:25.777221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:25.777396 (+ 175us) server_negotiation.cc:407] Connection header received
0925 22:41:25.777564 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:25.777567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:25.777608 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:25.777691 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:25.778743 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.779426 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.780240 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.780450 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.780561 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:25.783302 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:25.783310 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:25.783312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:25.783333 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:25.784879 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:25.785359 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:25.785364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:25.785366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:25.785441 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:25.785724 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:25.785727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:25.785728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:25.785907 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:25.786017 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:25.786295 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 22:41:25.786466 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":52}
I20260925 22:41:25.786654 10412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:25.776982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:38925 (local address 127.0.0.1:42840)
0925 22:41:25.777306 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:25.777317 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:25.777462 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:25.777776 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:25.777781 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:25.778470 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:25.778622 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:25.778629 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.779541 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.779545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:25.780117 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:25.780124 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.780565 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.780569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:25.780651 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:25.781299 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:25.781312 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:25.783151 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:25.785004 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:25.785014 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:25.785021 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:25.785257 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:25.785545 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:25.785547 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:25.785549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:25.785631 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:25.786017 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:25.786024 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:25.786137 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 22:41:25.786253 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 22:41:25.786524 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":114,"thread_start_us":133,"threads_started":1}
I20260925 22:41:25.791393 9860 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:42840:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 22:41:25.792285 9860 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:41:25.797777 9990 tablet_service.cc:1467] Processing CreateTablet for tablet 82591e3093b54005a5a872b61f1e6f66 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=208f2c8cc63f4683bdaee32d110c056d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:41:25.797827 10279 tablet_service.cc:1467] Processing CreateTablet for tablet 82591e3093b54005a5a872b61f1e6f66 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=208f2c8cc63f4683bdaee32d110c056d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:41:25.798204 9990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82591e3093b54005a5a872b61f1e6f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:25.798204 10279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82591e3093b54005a5a872b61f1e6f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:25.801242 10373 tablet_bootstrap.cc:492] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18: Bootstrap starting.
I20260925 22:41:25.801645 10139 tablet_service.cc:1467] Processing CreateTablet for tablet 82591e3093b54005a5a872b61f1e6f66 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=208f2c8cc63f4683bdaee32d110c056d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:41:25.801929 10139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82591e3093b54005a5a872b61f1e6f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:25.802282 10373 tablet_bootstrap.cc:654] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:25.803059 10374 tablet_bootstrap.cc:492] T 82591e3093b54005a5a872b61f1e6f66 P a7729df827f14e15aed7a83d71d537dd: Bootstrap starting.
I20260925 22:41:25.804080 10374 tablet_bootstrap.cc:654] T 82591e3093b54005a5a872b61f1e6f66 P a7729df827f14e15aed7a83d71d537dd: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:25.804450 10375 tablet_bootstrap.cc:492] T 82591e3093b54005a5a872b61f1e6f66 P 82f5b407bb024c43afec7bb9d5e25da5: Bootstrap starting.
I20260925 22:41:25.805392 10373 tablet_bootstrap.cc:492] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18: No bootstrap required, opened a new log
I20260925 22:41:25.805510 10373 ts_tablet_manager.cc:1397] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 22:41:25.805552 10375 tablet_bootstrap.cc:654] T 82591e3093b54005a5a872b61f1e6f66 P 82f5b407bb024c43afec7bb9d5e25da5: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:25.806061 10373 raft_consensus.cc:348] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:25.806195 10373 raft_consensus.cc:374] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:25.806223 10374 tablet_bootstrap.cc:492] T 82591e3093b54005a5a872b61f1e6f66 P a7729df827f14e15aed7a83d71d537dd: No bootstrap required, opened a new log
I20260925 22:41:25.806229 10373 raft_consensus.cc:729] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73b595951f524337992bb66a6d9e9b18, State: Initialized, Role: FOLLOWER
I20260925 22:41:25.806310 10374 ts_tablet_manager.cc:1397] T 82591e3093b54005a5a872b61f1e6f66 P a7729df827f14e15aed7a83d71d537dd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:41:25.806401 10373 consensus_queue.cc:260] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:25.806689 10373 ts_tablet_manager.cc:1428] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:41:25.806641 10379 tablet_replica.cc:439] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } } }
I20260925 22:41:25.806771 10379 tablet_replica.cc:442] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:41:25.806752 10374 raft_consensus.cc:348] T 82591e3093b54005a5a872b61f1e6f66 P a7729df827f14e15aed7a83d71d537dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:25.806878 10374 raft_consensus.cc:374] T 82591e3093b54005a5a872b61f1e6f66 P a7729df827f14e15aed7a83d71d537dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:25.806913 10374 raft_consensus.cc:729] T 82591e3093b54005a5a872b61f1e6f66 P a7729df827f14e15aed7a83d71d537dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7729df827f14e15aed7a83d71d537dd, State: Initialized, Role: FOLLOWER
I20260925 22:41:25.807067 10374 consensus_queue.cc:260] T 82591e3093b54005a5a872b61f1e6f66 P a7729df827f14e15aed7a83d71d537dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:25.807374 10374 ts_tablet_manager.cc:1428] T 82591e3093b54005a5a872b61f1e6f66 P a7729df827f14e15aed7a83d71d537dd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 22:41:25.808137 10375 tablet_bootstrap.cc:492] T 82591e3093b54005a5a872b61f1e6f66 P 82f5b407bb024c43afec7bb9d5e25da5: No bootstrap required, opened a new log
I20260925 22:41:25.808233 10375 ts_tablet_manager.cc:1397] T 82591e3093b54005a5a872b61f1e6f66 P 82f5b407bb024c43afec7bb9d5e25da5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 22:41:25.808298 10381 tablet_replica.cc:439] T 82591e3093b54005a5a872b61f1e6f66 P a7729df827f14e15aed7a83d71d537dd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } } }
I20260925 22:41:25.808482 10381 tablet_replica.cc:442] T 82591e3093b54005a5a872b61f1e6f66 P a7729df827f14e15aed7a83d71d537dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:41:25.808765 10375 raft_consensus.cc:348] T 82591e3093b54005a5a872b61f1e6f66 P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:25.808895 10375 raft_consensus.cc:374] T 82591e3093b54005a5a872b61f1e6f66 P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:25.808938 10375 raft_consensus.cc:729] T 82591e3093b54005a5a872b61f1e6f66 P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f5b407bb024c43afec7bb9d5e25da5, State: Initialized, Role: FOLLOWER
I20260925 22:41:25.809072 10375 consensus_queue.cc:260] T 82591e3093b54005a5a872b61f1e6f66 P 82f5b407bb024c43afec7bb9d5e25da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:25.809814 10392 tablet_replica.cc:439] T 82591e3093b54005a5a872b61f1e6f66 P 82f5b407bb024c43afec7bb9d5e25da5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } } }
I20260925 22:41:25.809998 10392 tablet_replica.cc:442] T 82591e3093b54005a5a872b61f1e6f66 P 82f5b407bb024c43afec7bb9d5e25da5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:41:25.810771 10375 ts_tablet_manager.cc:1428] T 82591e3093b54005a5a872b61f1e6f66 P 82f5b407bb024c43afec7bb9d5e25da5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 22:41:25.966151 10379 raft_consensus.cc:482] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:41:25.966349 10379 raft_consensus.cc:504] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:25.966822 10379 leader_election.cc:288] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82f5b407bb024c43afec7bb9d5e25da5 (127.2.74.194:44861), a7729df827f14e15aed7a83d71d537dd (127.2.74.195:45449)
I20260925 22:41:25.967485 10159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82591e3093b54005a5a872b61f1e6f66" candidate_uuid: "73b595951f524337992bb66a6d9e9b18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f5b407bb024c43afec7bb9d5e25da5" is_pre_election: true
I20260925 22:41:25.967460 10299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82591e3093b54005a5a872b61f1e6f66" candidate_uuid: "73b595951f524337992bb66a6d9e9b18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7729df827f14e15aed7a83d71d537dd" is_pre_election: true
I20260925 22:41:25.967641 10159 raft_consensus.cc:2454] T 82591e3093b54005a5a872b61f1e6f66 P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73b595951f524337992bb66a6d9e9b18 in term 0.
I20260925 22:41:25.967641 10299 raft_consensus.cc:2454] T 82591e3093b54005a5a872b61f1e6f66 P a7729df827f14e15aed7a83d71d537dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73b595951f524337992bb66a6d9e9b18 in term 0.
I20260925 22:41:25.967979 9953 leader_election.cc:302] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 73b595951f524337992bb66a6d9e9b18, 82f5b407bb024c43afec7bb9d5e25da5, a7729df827f14e15aed7a83d71d537dd; no voters:
I20260925 22:41:25.968194 10379 raft_consensus.cc:2781] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:41:25.968276 10379 raft_consensus.cc:482] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:41:25.968333 10379 raft_consensus.cc:3037] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:25.969326 10379 raft_consensus.cc:504] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:25.969736 10379 leader_election.cc:288] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18 [CANDIDATE]: Term 1 election: Requested vote from peers 82f5b407bb024c43afec7bb9d5e25da5 (127.2.74.194:44861), a7729df827f14e15aed7a83d71d537dd (127.2.74.195:45449)
I20260925 22:41:25.970479 10159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82591e3093b54005a5a872b61f1e6f66" candidate_uuid: "73b595951f524337992bb66a6d9e9b18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f5b407bb024c43afec7bb9d5e25da5"
I20260925 22:41:25.970479 10299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82591e3093b54005a5a872b61f1e6f66" candidate_uuid: "73b595951f524337992bb66a6d9e9b18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7729df827f14e15aed7a83d71d537dd"
I20260925 22:41:25.970639 10299 raft_consensus.cc:3037] T 82591e3093b54005a5a872b61f1e6f66 P a7729df827f14e15aed7a83d71d537dd [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:25.970639 10159 raft_consensus.cc:3037] T 82591e3093b54005a5a872b61f1e6f66 P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:25.971901 10159 raft_consensus.cc:2454] T 82591e3093b54005a5a872b61f1e6f66 P 82f5b407bb024c43afec7bb9d5e25da5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73b595951f524337992bb66a6d9e9b18 in term 1.
I20260925 22:41:25.971901 10299 raft_consensus.cc:2454] T 82591e3093b54005a5a872b61f1e6f66 P a7729df827f14e15aed7a83d71d537dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73b595951f524337992bb66a6d9e9b18 in term 1.
I20260925 22:41:25.972340 9953 leader_election.cc:302] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73b595951f524337992bb66a6d9e9b18, a7729df827f14e15aed7a83d71d537dd; no voters:
I20260925 22:41:25.972599 10379 raft_consensus.cc:2781] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:41:25.972697 10379 raft_consensus.cc:686] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18 [term 1 LEADER]: Becoming Leader. State: Replica: 73b595951f524337992bb66a6d9e9b18, State: Running, Role: LEADER
I20260925 22:41:25.972851 10379 consensus_queue.cc:237] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:25.973649 10405 tablet_replica.cc:439] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18: TxnStatusTablet state changed. Reason: New leader 73b595951f524337992bb66a6d9e9b18. Latest consensus state: current_term: 1 leader_uuid: "73b595951f524337992bb66a6d9e9b18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } } }
I20260925 22:41:25.973790 10405 tablet_replica.cc:442] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:41:25.974131 10417 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 22:41:25.974370 10417 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 22:41:25.974902 9861 catalog_manager.cc:5495] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73b595951f524337992bb66a6d9e9b18 (127.2.74.193). New cstate: current_term: 1 leader_uuid: "73b595951f524337992bb66a6d9e9b18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } health_report { overall_health: UNKNOWN } } }
I20260925 22:41:26.000559 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:25.996427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45544 (local address 127.2.74.193:43217)
0925 22:41:25.996562 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:25.996566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:25.996578 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:41:25.996694 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:25.996696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:25.996748 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:25.996831 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:41:25.997109 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.997630 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.998891 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.999210 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.999319 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:25.999693 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:41:26.000149 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:41:26.000244 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:26.000362 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 22:41:26.000419 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:41:26.000581 10412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:25.996297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:43217 (local address 127.0.0.1:45544)
0925 22:41:25.996436 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:25.996452 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:25.996557 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:25.996869 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:25.996872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:25.996882 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:41:25.997001 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:25.997007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.997734 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.997737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:25.998737 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:25.998747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:25.999331 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:25.999335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:25.999436 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:25.999498 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:41:26.000254 (+ 756us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:41:26.000268 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 22:41:26.000340 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 22:41:26.000437 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 22:41:26.003588 10299 raft_consensus.cc:1261] T 82591e3093b54005a5a872b61f1e6f66 P a7729df827f14e15aed7a83d71d537dd [term 1 FOLLOWER]: Refusing update from remote peer 73b595951f524337992bb66a6d9e9b18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:41:26.003767 10159 raft_consensus.cc:1261] T 82591e3093b54005a5a872b61f1e6f66 P 82f5b407bb024c43afec7bb9d5e25da5 [term 1 FOLLOWER]: Refusing update from remote peer 73b595951f524337992bb66a6d9e9b18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:41:26.004101 10405 consensus_queue.cc:1035] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:41:26.004381 10379 consensus_queue.cc:1035] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:41:26.005666 10381 tablet_replica.cc:439] T 82591e3093b54005a5a872b61f1e6f66 P a7729df827f14e15aed7a83d71d537dd: TxnStatusTablet state changed. Reason: New leader 73b595951f524337992bb66a6d9e9b18. Latest consensus state: current_term: 1 leader_uuid: "73b595951f524337992bb66a6d9e9b18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } } }
I20260925 22:41:26.005784 10381 tablet_replica.cc:442] T 82591e3093b54005a5a872b61f1e6f66 P a7729df827f14e15aed7a83d71d537dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:41:26.006078 10392 tablet_replica.cc:439] T 82591e3093b54005a5a872b61f1e6f66 P 82f5b407bb024c43afec7bb9d5e25da5: TxnStatusTablet state changed. Reason: New leader 73b595951f524337992bb66a6d9e9b18. Latest consensus state: current_term: 1 leader_uuid: "73b595951f524337992bb66a6d9e9b18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } } }
I20260925 22:41:26.006194 10392 tablet_replica.cc:442] T 82591e3093b54005a5a872b61f1e6f66 P 82f5b407bb024c43afec7bb9d5e25da5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:41:26.008394 10405 tablet_replica.cc:439] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "73b595951f524337992bb66a6d9e9b18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } } }
I20260925 22:41:26.008519 10405 tablet_replica.cc:442] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:41:26.008623 10405 tablet_replica.cc:439] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "73b595951f524337992bb66a6d9e9b18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } } }
I20260925 22:41:26.008679 10405 tablet_replica.cc:442] T 82591e3093b54005a5a872b61f1e6f66 P 73b595951f524337992bb66a6d9e9b18: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:41:26.009997 10392 tablet_replica.cc:439] T 82591e3093b54005a5a872b61f1e6f66 P 82f5b407bb024c43afec7bb9d5e25da5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "73b595951f524337992bb66a6d9e9b18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } } }
I20260925 22:41:26.010140 10392 tablet_replica.cc:442] T 82591e3093b54005a5a872b61f1e6f66 P 82f5b407bb024c43afec7bb9d5e25da5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:41:26.010353 10381 tablet_replica.cc:439] T 82591e3093b54005a5a872b61f1e6f66 P a7729df827f14e15aed7a83d71d537dd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "73b595951f524337992bb66a6d9e9b18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } } }
I20260925 22:41:26.010463 10381 tablet_replica.cc:442] T 82591e3093b54005a5a872b61f1e6f66 P a7729df827f14e15aed7a83d71d537dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:41:26.017937 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:26.014348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45560 (local address 127.2.74.193:43217)
0925 22:41:26.014480 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:26.014484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:26.014497 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:41:26.014635 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:26.014637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:26.014679 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:26.014743 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:41:26.015073 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:26.015583 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:26.016851 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:26.017080 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:26.017167 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:26.017468 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:41:26.017545 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:41:26.017601 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:26.017719 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 22:41:26.017768 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:41:26.030701 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:26.026785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:54643 (local address 127.2.74.193:43217)
0925 22:41:26.026929 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:26.026934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:26.027198 (+ 264us) server_negotiation.cc:407] Connection header received
0925 22:41:26.027344 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:26.027347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:26.027393 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:26.027482 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:41:26.027776 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:26.028269 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:26.029593 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:26.029803 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:26.029887 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:26.030235 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:41:26.030308 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:41:26.030362 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:26.030526 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 22:41:26.030578 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:41:26.030746 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:26.026730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:43217 (local address 127.2.74.193:54643)
0925 22:41:26.027086 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:26.027112 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:26.027241 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:26.027517 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:26.027520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:26.027527 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:41:26.027664 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:26.027670 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:26.028389 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:26.028392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:26.029454 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:26.029467 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:26.029922 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:26.029926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:26.030026 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:26.030092 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:41:26.030390 (+ 298us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:41:26.030396 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:41:26.030466 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 22:41:26.030550 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":125,"threads_started":1}
I20260925 22:41:26.037955 10159 raft_consensus.cc:1261] T ccc99a712aac4098afa8121e6aa55ca9 P 82f5b407bb024c43afec7bb9d5e25da5 [term 1 FOLLOWER]: Refusing update from remote peer 73b595951f524337992bb66a6d9e9b18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:41:26.037955 10299 raft_consensus.cc:1261] T ccc99a712aac4098afa8121e6aa55ca9 P a7729df827f14e15aed7a83d71d537dd [term 1 FOLLOWER]: Refusing update from remote peer 73b595951f524337992bb66a6d9e9b18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:41:26.038375 10422 consensus_queue.cc:1035] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:41:26.038563 10405 consensus_queue.cc:1035] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:41:26.085083 9860 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:41:26.085273 9860 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:41:26.087618 9860 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8178a160028d4173af41c7df23c6947b: Sending DeleteTablet for 3 replicas of tablet ccc99a712aac4098afa8121e6aa55ca9
I20260925 22:41:26.088327 9995 tablet_service.cc:1514] Processing DeleteTablet for tablet ccc99a712aac4098afa8121e6aa55ca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:26 UTC) from {username='oryx', principal='oryx/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:45530
I20260925 22:41:26.088488 10139 tablet_service.cc:1514] Processing DeleteTablet for tablet ccc99a712aac4098afa8121e6aa55ca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:26 UTC) from {username='oryx', principal='oryx/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:38928
I20260925 22:41:26.088693 10397 tablet_replica.cc:330] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18: stopping tablet replica
I20260925 22:41:26.088845 10397 raft_consensus.cc:2229] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:41:26.089188 10397 raft_consensus.cc:2258] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:41:26.089421 10398 tablet_replica.cc:330] T ccc99a712aac4098afa8121e6aa55ca9 P 82f5b407bb024c43afec7bb9d5e25da5: stopping tablet replica
I20260925 22:41:26.089573 10398 raft_consensus.cc:2229] T ccc99a712aac4098afa8121e6aa55ca9 P 82f5b407bb024c43afec7bb9d5e25da5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:41:26.089721 10398 raft_consensus.cc:2258] T ccc99a712aac4098afa8121e6aa55ca9 P 82f5b407bb024c43afec7bb9d5e25da5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:41:26.090572 10398 ts_tablet_manager.cc:1905] T ccc99a712aac4098afa8121e6aa55ca9 P 82f5b407bb024c43afec7bb9d5e25da5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:41:26.093106 10398 ts_tablet_manager.cc:1918] T ccc99a712aac4098afa8121e6aa55ca9 P 82f5b407bb024c43afec7bb9d5e25da5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:41:26.093230 10398 log.cc:1196] T ccc99a712aac4098afa8121e6aa55ca9 P 82f5b407bb024c43afec7bb9d5e25da5: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-1/wal/wals/ccc99a712aac4098afa8121e6aa55ca9
I20260925 22:41:26.093516 10398 ts_tablet_manager.cc:1939] T ccc99a712aac4098afa8121e6aa55ca9 P 82f5b407bb024c43afec7bb9d5e25da5: Deleting consensus metadata
I20260925 22:41:26.094427 9848 catalog_manager.cc:4841] TS 82f5b407bb024c43afec7bb9d5e25da5 (127.2.74.194:44861): tablet ccc99a712aac4098afa8121e6aa55ca9 (table test-table [id=758206a8f4a049c1890251391d55018d]) successfully deleted
I20260925 22:41:26.094767 10397 ts_tablet_manager.cc:1905] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:41:26.097043 10397 ts_tablet_manager.cc:1918] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:41:26.097153 10397 log.cc:1196] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-0/wal/wals/ccc99a712aac4098afa8121e6aa55ca9
I20260925 22:41:26.097419 10397 ts_tablet_manager.cc:1939] T ccc99a712aac4098afa8121e6aa55ca9 P 73b595951f524337992bb66a6d9e9b18: Deleting consensus metadata
I20260925 22:41:26.098229 9847 catalog_manager.cc:4841] TS 73b595951f524337992bb66a6d9e9b18 (127.2.74.193:43217): tablet ccc99a712aac4098afa8121e6aa55ca9 (table test-table [id=758206a8f4a049c1890251391d55018d]) successfully deleted
I20260925 22:41:26.098263 10279 tablet_service.cc:1514] Processing DeleteTablet for tablet ccc99a712aac4098afa8121e6aa55ca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:26 UTC) from {username='oryx', principal='oryx/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:50612
I20260925 22:41:26.098477 10399 tablet_replica.cc:330] T ccc99a712aac4098afa8121e6aa55ca9 P a7729df827f14e15aed7a83d71d537dd: stopping tablet replica
I20260925 22:41:26.098625 10399 raft_consensus.cc:2229] T ccc99a712aac4098afa8121e6aa55ca9 P a7729df827f14e15aed7a83d71d537dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:41:26.098767 10399 raft_consensus.cc:2258] T ccc99a712aac4098afa8121e6aa55ca9 P a7729df827f14e15aed7a83d71d537dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:41:26.100047 10399 ts_tablet_manager.cc:1905] T ccc99a712aac4098afa8121e6aa55ca9 P a7729df827f14e15aed7a83d71d537dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:41:26.103158 10399 ts_tablet_manager.cc:1918] T ccc99a712aac4098afa8121e6aa55ca9 P a7729df827f14e15aed7a83d71d537dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:41:26.103255 10399 log.cc:1196] T ccc99a712aac4098afa8121e6aa55ca9 P a7729df827f14e15aed7a83d71d537dd: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-2/wal/wals/ccc99a712aac4098afa8121e6aa55ca9
I20260925 22:41:26.103513 10399 ts_tablet_manager.cc:1939] T ccc99a712aac4098afa8121e6aa55ca9 P a7729df827f14e15aed7a83d71d537dd: Deleting consensus metadata
I20260925 22:41:26.104276 9849 catalog_manager.cc:4841] TS a7729df827f14e15aed7a83d71d537dd (127.2.74.195:45449): tablet ccc99a712aac4098afa8121e6aa55ca9 (table test-table [id=758206a8f4a049c1890251391d55018d]) successfully deleted
Sep 25 22:41:26 dist-test-slave-csnl krb5kdc[9817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:41:26 dist-test-slave-csnl krb5kdc[9817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:26 dist-test-slave-csnl krb5kdc[9817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 22:41:26.128592 9911 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:26.120811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42844 (local address 127.2.74.254:38925)
0925 22:41:26.120984 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:26.120989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:26.121304 (+ 315us) server_negotiation.cc:407] Connection header received
0925 22:41:26.121455 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:26.121458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:26.121506 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:26.121600 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:26.122541 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:26.123201 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:26.124415 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:26.124668 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:26.124755 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:26.128436 (+ 3681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42844: BlockingRecv error: recv got EOF from 127.0.0.1:42844 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 22:41:26 dist-test-slave-csnl krb5kdc[9817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM
I20260925 22:41:26.139890 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:26.130322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42848 (local address 127.2.74.254:38925)
0925 22:41:26.130430 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:26.130434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:26.130556 (+ 122us) server_negotiation.cc:407] Connection header received
0925 22:41:26.130714 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:26.130716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:26.130759 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:26.130829 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:26.131690 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:26.132210 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:26.133335 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:26.133578 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:26.133670 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:26.136531 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:26.136535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:26.136538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:26.136567 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:26.138116 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:26.138581 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:26.138585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:26.138587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:26.138673 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:26.138994 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:26.139000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:26.139001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:26.139211 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:26.139352 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:26.139602 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 22:41:26.139717 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 22:41:26.143114 9860 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:41:26.143596 9860 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:41:26.149068 9995 tablet_service.cc:1467] Processing CreateTablet for tablet 043fd6da8e6a431dad7d3375a716f228 (DEFAULT_TABLE table=test-table [id=1c4caa2ebd2146daa56190965580c3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:26.149151 10139 tablet_service.cc:1467] Processing CreateTablet for tablet 043fd6da8e6a431dad7d3375a716f228 (DEFAULT_TABLE table=test-table [id=1c4caa2ebd2146daa56190965580c3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:26.149461 9995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 043fd6da8e6a431dad7d3375a716f228. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:26.149451 10139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 043fd6da8e6a431dad7d3375a716f228. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:26.152021 10373 tablet_bootstrap.cc:492] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18: Bootstrap starting.
I20260925 22:41:26.152381 10279 tablet_service.cc:1467] Processing CreateTablet for tablet 043fd6da8e6a431dad7d3375a716f228 (DEFAULT_TABLE table=test-table [id=1c4caa2ebd2146daa56190965580c3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:26.152689 10279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 043fd6da8e6a431dad7d3375a716f228. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:26.153090 10373 tablet_bootstrap.cc:654] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:26.154203 10373 tablet_bootstrap.cc:492] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18: No bootstrap required, opened a new log
I20260925 22:41:26.154326 10373 ts_tablet_manager.cc:1397] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:41:26.154758 10373 raft_consensus.cc:348] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:26.154896 10373 raft_consensus.cc:374] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:26.154938 10373 raft_consensus.cc:729] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73b595951f524337992bb66a6d9e9b18, State: Initialized, Role: FOLLOWER
I20260925 22:41:26.155107 10373 consensus_queue.cc:260] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:26.155243 10374 tablet_bootstrap.cc:492] T 043fd6da8e6a431dad7d3375a716f228 P a7729df827f14e15aed7a83d71d537dd: Bootstrap starting.
I20260925 22:41:26.155961 10373 ts_tablet_manager.cc:1428] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260925 22:41:26.156237 10374 tablet_bootstrap.cc:654] T 043fd6da8e6a431dad7d3375a716f228 P a7729df827f14e15aed7a83d71d537dd: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:26.156445 10375 tablet_bootstrap.cc:492] T 043fd6da8e6a431dad7d3375a716f228 P 82f5b407bb024c43afec7bb9d5e25da5: Bootstrap starting.
I20260925 22:41:26.157339 10375 tablet_bootstrap.cc:654] T 043fd6da8e6a431dad7d3375a716f228 P 82f5b407bb024c43afec7bb9d5e25da5: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:26.157938 10374 tablet_bootstrap.cc:492] T 043fd6da8e6a431dad7d3375a716f228 P a7729df827f14e15aed7a83d71d537dd: No bootstrap required, opened a new log
I20260925 22:41:26.158027 10374 ts_tablet_manager.cc:1397] T 043fd6da8e6a431dad7d3375a716f228 P a7729df827f14e15aed7a83d71d537dd: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 22:41:26.158533 10374 raft_consensus.cc:348] T 043fd6da8e6a431dad7d3375a716f228 P a7729df827f14e15aed7a83d71d537dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:26.158671 10374 raft_consensus.cc:374] T 043fd6da8e6a431dad7d3375a716f228 P a7729df827f14e15aed7a83d71d537dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:26.158702 10374 raft_consensus.cc:729] T 043fd6da8e6a431dad7d3375a716f228 P a7729df827f14e15aed7a83d71d537dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7729df827f14e15aed7a83d71d537dd, State: Initialized, Role: FOLLOWER
I20260925 22:41:26.158833 10374 consensus_queue.cc:260] T 043fd6da8e6a431dad7d3375a716f228 P a7729df827f14e15aed7a83d71d537dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:26.159579 10374 ts_tablet_manager.cc:1428] T 043fd6da8e6a431dad7d3375a716f228 P a7729df827f14e15aed7a83d71d537dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260925 22:41:26.159796 10375 tablet_bootstrap.cc:492] T 043fd6da8e6a431dad7d3375a716f228 P 82f5b407bb024c43afec7bb9d5e25da5: No bootstrap required, opened a new log
I20260925 22:41:26.159914 10375 ts_tablet_manager.cc:1397] T 043fd6da8e6a431dad7d3375a716f228 P 82f5b407bb024c43afec7bb9d5e25da5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 22:41:26.160362 10375 raft_consensus.cc:348] T 043fd6da8e6a431dad7d3375a716f228 P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:26.160512 10375 raft_consensus.cc:374] T 043fd6da8e6a431dad7d3375a716f228 P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:26.160545 10375 raft_consensus.cc:729] T 043fd6da8e6a431dad7d3375a716f228 P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f5b407bb024c43afec7bb9d5e25da5, State: Initialized, Role: FOLLOWER
I20260925 22:41:26.160676 10375 consensus_queue.cc:260] T 043fd6da8e6a431dad7d3375a716f228 P 82f5b407bb024c43afec7bb9d5e25da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:26.161003 10375 ts_tablet_manager.cc:1428] T 043fd6da8e6a431dad7d3375a716f228 P 82f5b407bb024c43afec7bb9d5e25da5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:41:26.206719 10405 raft_consensus.cc:482] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:41:26.206882 10405 raft_consensus.cc:504] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:26.207378 10405 leader_election.cc:288] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82f5b407bb024c43afec7bb9d5e25da5 (127.2.74.194:44861), a7729df827f14e15aed7a83d71d537dd (127.2.74.195:45449)
I20260925 22:41:26.207798 10159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043fd6da8e6a431dad7d3375a716f228" candidate_uuid: "73b595951f524337992bb66a6d9e9b18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f5b407bb024c43afec7bb9d5e25da5" is_pre_election: true
I20260925 22:41:26.207839 10299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043fd6da8e6a431dad7d3375a716f228" candidate_uuid: "73b595951f524337992bb66a6d9e9b18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7729df827f14e15aed7a83d71d537dd" is_pre_election: true
I20260925 22:41:26.207962 10159 raft_consensus.cc:2454] T 043fd6da8e6a431dad7d3375a716f228 P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73b595951f524337992bb66a6d9e9b18 in term 0.
I20260925 22:41:26.207989 10299 raft_consensus.cc:2454] T 043fd6da8e6a431dad7d3375a716f228 P a7729df827f14e15aed7a83d71d537dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73b595951f524337992bb66a6d9e9b18 in term 0.
I20260925 22:41:26.208308 9952 leader_election.cc:302] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73b595951f524337992bb66a6d9e9b18, 82f5b407bb024c43afec7bb9d5e25da5; no voters:
I20260925 22:41:26.208484 10405 raft_consensus.cc:2781] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:41:26.208547 10405 raft_consensus.cc:482] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:41:26.208607 10405 raft_consensus.cc:3037] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:26.209825 10405 raft_consensus.cc:504] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:26.210245 10405 leader_election.cc:288] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18 [CANDIDATE]: Term 1 election: Requested vote from peers 82f5b407bb024c43afec7bb9d5e25da5 (127.2.74.194:44861), a7729df827f14e15aed7a83d71d537dd (127.2.74.195:45449)
I20260925 22:41:26.210682 10299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043fd6da8e6a431dad7d3375a716f228" candidate_uuid: "73b595951f524337992bb66a6d9e9b18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7729df827f14e15aed7a83d71d537dd"
I20260925 22:41:26.210827 10299 raft_consensus.cc:3037] T 043fd6da8e6a431dad7d3375a716f228 P a7729df827f14e15aed7a83d71d537dd [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:26.211122 10159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043fd6da8e6a431dad7d3375a716f228" candidate_uuid: "73b595951f524337992bb66a6d9e9b18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f5b407bb024c43afec7bb9d5e25da5"
I20260925 22:41:26.211265 10159 raft_consensus.cc:3037] T 043fd6da8e6a431dad7d3375a716f228 P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:26.212044 10299 raft_consensus.cc:2454] T 043fd6da8e6a431dad7d3375a716f228 P a7729df827f14e15aed7a83d71d537dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73b595951f524337992bb66a6d9e9b18 in term 1.
I20260925 22:41:26.212248 10159 raft_consensus.cc:2454] T 043fd6da8e6a431dad7d3375a716f228 P 82f5b407bb024c43afec7bb9d5e25da5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73b595951f524337992bb66a6d9e9b18 in term 1.
I20260925 22:41:26.212435 9953 leader_election.cc:302] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73b595951f524337992bb66a6d9e9b18, a7729df827f14e15aed7a83d71d537dd; no voters:
I20260925 22:41:26.212749 10405 raft_consensus.cc:2781] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:41:26.212858 10405 raft_consensus.cc:686] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18 [term 1 LEADER]: Becoming Leader. State: Replica: 73b595951f524337992bb66a6d9e9b18, State: Running, Role: LEADER
I20260925 22:41:26.213003 10405 consensus_queue.cc:237] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:26.214867 9861 catalog_manager.cc:5495] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73b595951f524337992bb66a6d9e9b18 (127.2.74.193). New cstate: current_term: 1 leader_uuid: "73b595951f524337992bb66a6d9e9b18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } health_report { overall_health: UNKNOWN } } }
I20260925 22:41:26.226275 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:26.223049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45576 (local address 127.2.74.193:43217)
0925 22:41:26.223203 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:26.223208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:26.223222 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:41:26.223354 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:26.223357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:26.223413 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:26.223495 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:41:26.223744 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:26.224252 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:26.225158 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:26.225378 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:26.225488 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:26.225763 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:41:26.225842 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:41:26.225921 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:26.226012 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 22:41:26.226066 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:41:26.228729 10159 raft_consensus.cc:1261] T 043fd6da8e6a431dad7d3375a716f228 P 82f5b407bb024c43afec7bb9d5e25da5 [term 1 FOLLOWER]: Refusing update from remote peer 73b595951f524337992bb66a6d9e9b18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:41:26.229182 10299 raft_consensus.cc:1261] T 043fd6da8e6a431dad7d3375a716f228 P a7729df827f14e15aed7a83d71d537dd [term 1 FOLLOWER]: Refusing update from remote peer 73b595951f524337992bb66a6d9e9b18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:41:26.229202 10405 consensus_queue.cc:1035] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:41:26.229671 10405 consensus_queue.cc:1035] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:41:26.236413 9861 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:41:26.236583 9861 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:41:26.239269 9861 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8178a160028d4173af41c7df23c6947b: Sending DeleteTablet for 3 replicas of tablet 043fd6da8e6a431dad7d3375a716f228
I20260925 22:41:26.239832 9995 tablet_service.cc:1514] Processing DeleteTablet for tablet 043fd6da8e6a431dad7d3375a716f228 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:26 UTC) from {username='oryx', principal='oryx/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:45530
I20260925 22:41:26.239866 10139 tablet_service.cc:1514] Processing DeleteTablet for tablet 043fd6da8e6a431dad7d3375a716f228 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:26 UTC) from {username='oryx', principal='oryx/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:38928
I20260925 22:41:26.240037 10398 tablet_replica.cc:330] T 043fd6da8e6a431dad7d3375a716f228 P 82f5b407bb024c43afec7bb9d5e25da5: stopping tablet replica
I20260925 22:41:26.240036 10397 tablet_replica.cc:330] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18: stopping tablet replica
I20260925 22:41:26.240155 10398 raft_consensus.cc:2229] T 043fd6da8e6a431dad7d3375a716f228 P 82f5b407bb024c43afec7bb9d5e25da5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:41:26.240154 10397 raft_consensus.cc:2229] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:41:26.240286 10398 raft_consensus.cc:2258] T 043fd6da8e6a431dad7d3375a716f228 P 82f5b407bb024c43afec7bb9d5e25da5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:41:26.240358 10397 raft_consensus.cc:2258] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:41:26.240856 10279 tablet_service.cc:1514] Processing DeleteTablet for tablet 043fd6da8e6a431dad7d3375a716f228 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:26 UTC) from {username='oryx', principal='oryx/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:50612
I20260925 22:41:26.241083 10399 tablet_replica.cc:330] T 043fd6da8e6a431dad7d3375a716f228 P a7729df827f14e15aed7a83d71d537dd: stopping tablet replica
I20260925 22:41:26.241204 10399 raft_consensus.cc:2229] T 043fd6da8e6a431dad7d3375a716f228 P a7729df827f14e15aed7a83d71d537dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:41:26.241328 10399 raft_consensus.cc:2258] T 043fd6da8e6a431dad7d3375a716f228 P a7729df827f14e15aed7a83d71d537dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:41:26.241454 10397 ts_tablet_manager.cc:1905] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:41:26.242007 10399 ts_tablet_manager.cc:1905] T 043fd6da8e6a431dad7d3375a716f228 P a7729df827f14e15aed7a83d71d537dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:41:26.242394 9861 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:41:26.242787 9861 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:41:26.244333 10397 ts_tablet_manager.cc:1918] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:41:26.244406 10397 log.cc:1196] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-0/wal/wals/043fd6da8e6a431dad7d3375a716f228
I20260925 22:41:26.244679 10397 ts_tablet_manager.cc:1939] T 043fd6da8e6a431dad7d3375a716f228 P 73b595951f524337992bb66a6d9e9b18: Deleting consensus metadata
I20260925 22:41:26.244690 10399 ts_tablet_manager.cc:1918] T 043fd6da8e6a431dad7d3375a716f228 P a7729df827f14e15aed7a83d71d537dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:41:26.244755 10399 log.cc:1196] T 043fd6da8e6a431dad7d3375a716f228 P a7729df827f14e15aed7a83d71d537dd: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-2/wal/wals/043fd6da8e6a431dad7d3375a716f228
I20260925 22:41:26.244992 10399 ts_tablet_manager.cc:1939] T 043fd6da8e6a431dad7d3375a716f228 P a7729df827f14e15aed7a83d71d537dd: Deleting consensus metadata
I20260925 22:41:26.245409 9847 catalog_manager.cc:4841] TS 73b595951f524337992bb66a6d9e9b18 (127.2.74.193:43217): tablet 043fd6da8e6a431dad7d3375a716f228 (table test-table [id=1c4caa2ebd2146daa56190965580c3c0]) successfully deleted
I20260925 22:41:26.245788 9849 catalog_manager.cc:4841] TS a7729df827f14e15aed7a83d71d537dd (127.2.74.195:45449): tablet 043fd6da8e6a431dad7d3375a716f228 (table test-table [id=1c4caa2ebd2146daa56190965580c3c0]) successfully deleted
I20260925 22:41:26.246120 10398 ts_tablet_manager.cc:1905] T 043fd6da8e6a431dad7d3375a716f228 P 82f5b407bb024c43afec7bb9d5e25da5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:41:26.248076 10279 tablet_service.cc:1467] Processing CreateTablet for tablet 2b6d57f52a3d43a5ba21c68c4afa1c0c (DEFAULT_TABLE table=test-table [id=9e8afc3ad80145d7adae6d4c6a44005f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:26.248270 10139 tablet_service.cc:1467] Processing CreateTablet for tablet 2b6d57f52a3d43a5ba21c68c4afa1c0c (DEFAULT_TABLE table=test-table [id=9e8afc3ad80145d7adae6d4c6a44005f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:26.248394 10279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b6d57f52a3d43a5ba21c68c4afa1c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:26.248423 10398 ts_tablet_manager.cc:1918] T 043fd6da8e6a431dad7d3375a716f228 P 82f5b407bb024c43afec7bb9d5e25da5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:41:26.248484 10398 log.cc:1196] T 043fd6da8e6a431dad7d3375a716f228 P 82f5b407bb024c43afec7bb9d5e25da5: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-1/wal/wals/043fd6da8e6a431dad7d3375a716f228
I20260925 22:41:26.248525 10139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b6d57f52a3d43a5ba21c68c4afa1c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:26.248768 10398 ts_tablet_manager.cc:1939] T 043fd6da8e6a431dad7d3375a716f228 P 82f5b407bb024c43afec7bb9d5e25da5: Deleting consensus metadata
I20260925 22:41:26.250577 10374 tablet_bootstrap.cc:492] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P a7729df827f14e15aed7a83d71d537dd: Bootstrap starting.
I20260925 22:41:26.250696 10375 tablet_bootstrap.cc:492] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5: Bootstrap starting.
I20260925 22:41:26.251531 10375 tablet_bootstrap.cc:654] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:26.251786 10374 tablet_bootstrap.cc:654] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P a7729df827f14e15aed7a83d71d537dd: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:26.252236 9848 catalog_manager.cc:4841] TS 82f5b407bb024c43afec7bb9d5e25da5 (127.2.74.194:44861): tablet 043fd6da8e6a431dad7d3375a716f228 (table test-table [id=1c4caa2ebd2146daa56190965580c3c0]) successfully deleted
I20260925 22:41:26.252964 10374 tablet_bootstrap.cc:492] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P a7729df827f14e15aed7a83d71d537dd: No bootstrap required, opened a new log
I20260925 22:41:26.253043 10374 ts_tablet_manager.cc:1397] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P a7729df827f14e15aed7a83d71d537dd: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 22:41:26.253191 9995 tablet_service.cc:1467] Processing CreateTablet for tablet 2b6d57f52a3d43a5ba21c68c4afa1c0c (DEFAULT_TABLE table=test-table [id=9e8afc3ad80145d7adae6d4c6a44005f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:26.253340 10375 tablet_bootstrap.cc:492] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5: No bootstrap required, opened a new log
I20260925 22:41:26.253410 10375 ts_tablet_manager.cc:1397] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 22:41:26.253439 9995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b6d57f52a3d43a5ba21c68c4afa1c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:26.253507 10374 raft_consensus.cc:348] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P a7729df827f14e15aed7a83d71d537dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:26.253611 10374 raft_consensus.cc:374] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P a7729df827f14e15aed7a83d71d537dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:26.253643 10374 raft_consensus.cc:729] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P a7729df827f14e15aed7a83d71d537dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7729df827f14e15aed7a83d71d537dd, State: Initialized, Role: FOLLOWER
I20260925 22:41:26.253767 10374 consensus_queue.cc:260] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P a7729df827f14e15aed7a83d71d537dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:26.253774 10375 raft_consensus.cc:348] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:26.253877 10375 raft_consensus.cc:374] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:26.253906 10375 raft_consensus.cc:729] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f5b407bb024c43afec7bb9d5e25da5, State: Initialized, Role: FOLLOWER
I20260925 22:41:26.254015 10374 ts_tablet_manager.cc:1428] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P a7729df827f14e15aed7a83d71d537dd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 22:41:26.254022 10375 consensus_queue.cc:260] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:26.254256 10375 ts_tablet_manager.cc:1428] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:41:26.255875 10373 tablet_bootstrap.cc:492] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 73b595951f524337992bb66a6d9e9b18: Bootstrap starting.
I20260925 22:41:26.256672 10373 tablet_bootstrap.cc:654] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 73b595951f524337992bb66a6d9e9b18: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:26.257789 10373 tablet_bootstrap.cc:492] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 73b595951f524337992bb66a6d9e9b18: No bootstrap required, opened a new log
I20260925 22:41:26.257911 10373 ts_tablet_manager.cc:1397] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 73b595951f524337992bb66a6d9e9b18: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 22:41:26.258260 10373 raft_consensus.cc:348] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:26.258381 10373 raft_consensus.cc:374] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:26.258440 10373 raft_consensus.cc:729] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73b595951f524337992bb66a6d9e9b18, State: Initialized, Role: FOLLOWER
I20260925 22:41:26.258595 10373 consensus_queue.cc:260] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 73b595951f524337992bb66a6d9e9b18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:26.258879 10373 ts_tablet_manager.cc:1428] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 73b595951f524337992bb66a6d9e9b18: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:41:26.337880 10392 raft_consensus.cc:482] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:41:26.338040 10392 raft_consensus.cc:504] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:26.338469 10392 leader_election.cc:288] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73b595951f524337992bb66a6d9e9b18 (127.2.74.193:43217), a7729df827f14e15aed7a83d71d537dd (127.2.74.195:45449)
I20260925 22:41:26.339018 10006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6d57f52a3d43a5ba21c68c4afa1c0c" candidate_uuid: "82f5b407bb024c43afec7bb9d5e25da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73b595951f524337992bb66a6d9e9b18" is_pre_election: true
I20260925 22:41:26.339047 10299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6d57f52a3d43a5ba21c68c4afa1c0c" candidate_uuid: "82f5b407bb024c43afec7bb9d5e25da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7729df827f14e15aed7a83d71d537dd" is_pre_election: true
I20260925 22:41:26.339184 10006 raft_consensus.cc:2454] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82f5b407bb024c43afec7bb9d5e25da5 in term 0.
I20260925 22:41:26.339216 10299 raft_consensus.cc:2454] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P a7729df827f14e15aed7a83d71d537dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82f5b407bb024c43afec7bb9d5e25da5 in term 0.
I20260925 22:41:26.339509 10091 leader_election.cc:302] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73b595951f524337992bb66a6d9e9b18, 82f5b407bb024c43afec7bb9d5e25da5; no voters:
I20260925 22:41:26.339702 10392 raft_consensus.cc:2781] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:41:26.339785 10392 raft_consensus.cc:482] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:41:26.339839 10392 raft_consensus.cc:3037] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:26.341238 10392 raft_consensus.cc:504] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:26.341679 10392 leader_election.cc:288] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5 [CANDIDATE]: Term 1 election: Requested vote from peers 73b595951f524337992bb66a6d9e9b18 (127.2.74.193:43217), a7729df827f14e15aed7a83d71d537dd (127.2.74.195:45449)
I20260925 22:41:26.341970 10006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6d57f52a3d43a5ba21c68c4afa1c0c" candidate_uuid: "82f5b407bb024c43afec7bb9d5e25da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73b595951f524337992bb66a6d9e9b18"
I20260925 22:41:26.342096 10006 raft_consensus.cc:3037] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 73b595951f524337992bb66a6d9e9b18 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:26.342159 10299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6d57f52a3d43a5ba21c68c4afa1c0c" candidate_uuid: "82f5b407bb024c43afec7bb9d5e25da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7729df827f14e15aed7a83d71d537dd"
I20260925 22:41:26.342312 10299 raft_consensus.cc:3037] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P a7729df827f14e15aed7a83d71d537dd [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:26.343313 10299 raft_consensus.cc:2454] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P a7729df827f14e15aed7a83d71d537dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82f5b407bb024c43afec7bb9d5e25da5 in term 1.
I20260925 22:41:26.343324 10006 raft_consensus.cc:2454] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 73b595951f524337992bb66a6d9e9b18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82f5b407bb024c43afec7bb9d5e25da5 in term 1.
I20260925 22:41:26.343672 10093 leader_election.cc:302] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82f5b407bb024c43afec7bb9d5e25da5, a7729df827f14e15aed7a83d71d537dd; no voters:
I20260925 22:41:26.343904 10392 raft_consensus.cc:2781] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:41:26.344123 10392 raft_consensus.cc:686] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5 [term 1 LEADER]: Becoming Leader. State: Replica: 82f5b407bb024c43afec7bb9d5e25da5, State: Running, Role: LEADER
I20260925 22:41:26.344281 10392 consensus_queue.cc:237] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } }
I20260925 22:41:26.346069 9861 catalog_manager.cc:5495] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82f5b407bb024c43afec7bb9d5e25da5 (127.2.74.194). New cstate: current_term: 1 leader_uuid: "82f5b407bb024c43afec7bb9d5e25da5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82f5b407bb024c43afec7bb9d5e25da5" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 } health_report { overall_health: UNKNOWN } } }
I20260925 22:41:26.373490 10453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:26.370200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38950 (local address 127.2.74.194:44861)
0925 22:41:26.370452 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:26.370457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:26.370477 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:41:26.370518 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:26.370520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:26.370561 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:26.370636 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:41:26.370889 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:26.371576 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:26.372385 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:26.372648 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:26.372740 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:26.372961 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:41:26.373032 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:41:26.373090 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:26.373200 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 22:41:26.373247 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":98,"threads_started":1}
I20260925 22:41:26.384366 10456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:26.380592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:43217 (local address 127.2.74.194:53565)
0925 22:41:26.380892 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:26.380906 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:26.381015 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:26.381308 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:26.381312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:26.381323 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:41:26.381437 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:26.381442 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:26.382300 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:26.382302 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:26.383080 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:26.383090 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:26.383661 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:26.383664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:26.383726 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:26.383779 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:41:26.384104 (+ 325us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:41:26.384110 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:41:26.384162 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 22:41:26.384235 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":105,"threads_started":1}
I20260925 22:41:26.384383 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:26.380674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:53565 (local address 127.2.74.193:43217)
0925 22:41:26.380827 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:26.380831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:26.380979 (+ 148us) server_negotiation.cc:407] Connection header received
0925 22:41:26.381127 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:26.381130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:26.381185 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:26.381261 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:41:26.381557 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:26.382155 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:26.383226 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:26.383541 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:26.383629 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:26.383942 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:41:26.384028 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:41:26.384083 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:26.384187 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 22:41:26.384235 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 22:41:26.390470 10006 raft_consensus.cc:1261] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 73b595951f524337992bb66a6d9e9b18 [term 1 FOLLOWER]: Refusing update from remote peer 82f5b407bb024c43afec7bb9d5e25da5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:41:26.390460 10299 raft_consensus.cc:1261] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P a7729df827f14e15aed7a83d71d537dd [term 1 FOLLOWER]: Refusing update from remote peer 82f5b407bb024c43afec7bb9d5e25da5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:41:26.390866 10392 consensus_queue.cc:1035] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7729df827f14e15aed7a83d71d537dd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 45449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:41:26.391037 10451 consensus_queue.cc:1035] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73b595951f524337992bb66a6d9e9b18" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:41:26.407786 10453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:26.404571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:40505 (local address 127.2.74.194:44861)
0925 22:41:26.404680 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:26.404685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:26.404704 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:41:26.404785 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:26.404787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:26.404829 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:26.404921 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:41:26.405191 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:26.405928 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:26.406736 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:26.406997 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:26.407092 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:26.407323 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:41:26.407432 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:41:26.407508 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:26.407594 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 22:41:26.407672 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 22:41:26.407843 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:26.404504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:44861 (local address 127.2.74.193:40505)
0925 22:41:26.404623 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:26.404639 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:26.404715 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:26.404933 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:26.404936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:26.404943 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:41:26.405083 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:26.405089 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:26.406034 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:26.406037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:26.406629 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:26.406638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:26.407109 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:26.407112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:26.407174 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:26.407204 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:41:26.407520 (+ 316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:41:26.407526 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:41:26.407603 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 22:41:26.407706 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 22:41:26.427047 9861 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:41:26.427217 9861 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:41:26.429355 9861 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8178a160028d4173af41c7df23c6947b: Sending DeleteTablet for 3 replicas of tablet 2b6d57f52a3d43a5ba21c68c4afa1c0c
I20260925 22:41:26.430020 10139 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b6d57f52a3d43a5ba21c68c4afa1c0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:26 UTC) from {username='oryx', principal='oryx/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:38928
I20260925 22:41:26.430127 10279 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b6d57f52a3d43a5ba21c68c4afa1c0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:26 UTC) from {username='oryx', principal='oryx/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:50612
I20260925 22:41:26.430307 10399 tablet_replica.cc:330] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P a7729df827f14e15aed7a83d71d537dd: stopping tablet replica
I20260925 22:41:26.430306 10398 tablet_replica.cc:330] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5: stopping tablet replica
I20260925 22:41:26.430419 10398 raft_consensus.cc:2229] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:41:26.430419 10399 raft_consensus.cc:2229] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P a7729df827f14e15aed7a83d71d537dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:41:26.430534 10399 raft_consensus.cc:2258] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P a7729df827f14e15aed7a83d71d537dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:41:26.430650 10398 raft_consensus.cc:2258] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:41:26.430686 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 9934
I20260925 22:41:26.432543 10399 ts_tablet_manager.cc:1905] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P a7729df827f14e15aed7a83d71d537dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:41:26.435176 10399 ts_tablet_manager.cc:1918] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P a7729df827f14e15aed7a83d71d537dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:41:26.435281 10399 log.cc:1196] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P a7729df827f14e15aed7a83d71d537dd: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-2/wal/wals/2b6d57f52a3d43a5ba21c68c4afa1c0c
I20260925 22:41:26.435529 10398 ts_tablet_manager.cc:1905] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:41:26.435601 10399 ts_tablet_manager.cc:1939] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P a7729df827f14e15aed7a83d71d537dd: Deleting consensus metadata
I20260925 22:41:26.436354 9849 catalog_manager.cc:4841] TS a7729df827f14e15aed7a83d71d537dd (127.2.74.195:45449): tablet 2b6d57f52a3d43a5ba21c68c4afa1c0c (table test-table [id=9e8afc3ad80145d7adae6d4c6a44005f]) successfully deleted
I20260925 22:41:26.437569 10398 ts_tablet_manager.cc:1918] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:41:26.437642 10398 log.cc:1196] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376047580784-2347-0/minicluster-data/ts-1/wal/wals/2b6d57f52a3d43a5ba21c68c4afa1c0c
I20260925 22:41:26.437928 10398 ts_tablet_manager.cc:1939] T 2b6d57f52a3d43a5ba21c68c4afa1c0c P 82f5b407bb024c43afec7bb9d5e25da5: Deleting consensus metadata
I20260925 22:41:26.438755 9848 catalog_manager.cc:4841] TS 82f5b407bb024c43afec7bb9d5e25da5 (127.2.74.194:44861): tablet 2b6d57f52a3d43a5ba21c68c4afa1c0c (table test-table [id=9e8afc3ad80145d7adae6d4c6a44005f]) successfully deleted
W20260925 22:41:26.443590 9847 connection.cc:535] client connection to 127.2.74.193:43217 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 22:41:26.443837 9847 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 22:41:26.443918 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 10074
I20260925 22:41:26.445864 10461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:26.445121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:43217 (local address 127.0.0.1:45586)
0925 22:41:26.445643 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:26.445731 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.193:43217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":72,"threads_started":1}
W20260925 22:41:26.446003 9847 catalog_manager.cc:4584] TS 73b595951f524337992bb66a6d9e9b18 (127.2.74.193:43217): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b6d57f52a3d43a5ba21c68c4afa1c0c: Network error: Client connection negotiation failed: client connection to 127.2.74.193:43217: connect: Connection refused (error 111)
I20260925 22:41:26.451349 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 10214
I20260925 22:41:26.457329 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 9830
2026-09-25T22:41:26Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3261 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:41:26 dist-test-slave-csnl krb5kdc[10464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:41:26 dist-test-slave-csnl krb5kdc[10464](info): set up 2 sockets
Sep 25 22:41:26 dist-test-slave-csnl krb5kdc[10464](info): commencing operation
krb5kdc: starting...
W20260925 22:41:27.495929 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:41:27 dist-test-slave-csnl krb5kdc[10464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:41:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:41:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:41:27.622313 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:37709
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:42077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:37709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:27.736176 10477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:27.736544 10477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:27.736627 10477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:27.740924 10477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:41:27.741050 10477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:27.741088 10477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:27.741127 10477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:41:27.741161 10477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:41:27.746843 10477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:42077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:37709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:37709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:27.748121 10477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:27.749285 10477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:27.758773 10483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:27.758687 10482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:27.758699 10485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:27.758710 10477 server_base.cc:1034] running on GCE node
I20260925 22:41:27.759635 10477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:27.760921 10477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:27.762102 10477 hybrid_clock.cc:648] HybridClock initialized: now 1790376087762091 us; error 32 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T22:41:27Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1310 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:41:27 dist-test-slave-csnl krb5kdc[10490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:41:27 dist-test-slave-csnl krb5kdc[10490](info): set up 2 sockets
Sep 25 22:41:27 dist-test-slave-csnl krb5kdc[10490](info): commencing operation
krb5kdc: starting...
W20260925 22:41:28.823745 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:41:28 dist-test-slave-csnl krb5kdc[10490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:41:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:41:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:41:28.956192 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:41871
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:37347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:41871
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:29.071780 10503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:29.072103 10503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:29.072147 10503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:29.076927 10503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:41:29.077039 10503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:29.077060 10503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:29.077078 10503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:41:29.077095 10503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:41:29.083242 10503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:37347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:41871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:41871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:29.084571 10503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:29.085737 10503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:29.095052 10509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:29.095165 10503 server_base.cc:1034] running on GCE node
W20260925 22:41:29.095045 10508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:29.095043 10511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:29.096011 10503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:29.097295 10503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:29.098511 10503 hybrid_clock.cc:648] HybridClock initialized: now 1790376089098481 us; error 62 us; skew 500 ppm
Sep 25 22:41:29 dist-test-slave-csnl krb5kdc[10490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376089, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:29.101559 10503 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:41:29.102524 10503 webserver.cc:460] Webserver started at http://127.2.74.254:45823/ using document root <none> and password file <none>
I20260925 22:41:29.103199 10503 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:29.103295 10503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:29.103540 10503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:29.105716 10503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "66eeecb9e32f424a8fdd913b908b4a52"
format_stamp: "Formatted at 2026-09-25 22:41:29 on dist-test-slave-csnl"
server_key: "1ef2ddd5c70992c1beacbe0fd5b76f0b"
server_key_iv: "5a9eaac066befa58b72fabf2284bfc55"
server_key_version: "encryptionkey@0"
I20260925 22:41:29.106352 10503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "66eeecb9e32f424a8fdd913b908b4a52"
format_stamp: "Formatted at 2026-09-25 22:41:29 on dist-test-slave-csnl"
server_key: "1ef2ddd5c70992c1beacbe0fd5b76f0b"
server_key_iv: "5a9eaac066befa58b72fabf2284bfc55"
server_key_version: "encryptionkey@0"
I20260925 22:41:29.110435 10503 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:41:29.113195 10518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:29.114579 10503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 22:41:29.114746 10503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "66eeecb9e32f424a8fdd913b908b4a52"
format_stamp: "Formatted at 2026-09-25 22:41:29 on dist-test-slave-csnl"
server_key: "1ef2ddd5c70992c1beacbe0fd5b76f0b"
server_key_iv: "5a9eaac066befa58b72fabf2284bfc55"
server_key_version: "encryptionkey@0"
I20260925 22:41:29.114880 10503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:29.137905 10503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:29.141734 10503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:29.141978 10503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:29.149771 10503 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:41871
I20260925 22:41:29.149788 10570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:41871 every 8 connection(s)
I20260925 22:41:29.150875 10503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:41:29.153424 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 10503
I20260925 22:41:29.153594 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:41:29.153900 2347 external_mini_cluster.cc:1428] Setting key 34d8f7ffed23b8eb94869425ff9d4521
I20260925 22:41:29.154274 10571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:29.161144 10571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52: Bootstrap starting.
Sep 25 22:41:29 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:41:29.164002 10571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:29.164836 10571 log.cc:824] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:29.167372 10571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52: No bootstrap required, opened a new log
I20260925 22:41:29.169623 10574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:29.155571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43588 (local address 127.2.74.254:41871)
0925 22:41:29.156064 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:29.156084 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:29.156124 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:41:29.156816 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:29.156835 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:29.157198 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:29.157595 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:29.158438 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:29.159596 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:29.160509 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:29.161020 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:29.161213 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:29.164202 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:29.164209 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:29.164226 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:29.164263 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:29.166335 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:29.166907 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:29.166913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:29.166918 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:29.167111 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:29.167442 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:29.167445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:29.167447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:29.167956 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:29.168175 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:29.168500 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 22:41:29.168840 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":148,"threads_started":1}
I20260925 22:41:29.171974 10571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66eeecb9e32f424a8fdd913b908b4a52" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 41871 } }
I20260925 22:41:29.172346 10571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:29.172438 10571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66eeecb9e32f424a8fdd913b908b4a52, State: Initialized, Role: FOLLOWER
I20260925 22:41:29.172946 10571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66eeecb9e32f424a8fdd913b908b4a52" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 41871 } }
I20260925 22:41:29.173120 10571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:41:29.173218 10571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:41:29.173354 10571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:29.174443 10571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66eeecb9e32f424a8fdd913b908b4a52" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 41871 } }
I20260925 22:41:29.174853 10571 leader_election.cc:302] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66eeecb9e32f424a8fdd913b908b4a52; no voters:
I20260925 22:41:29.175262 10571 leader_election.cc:288] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:41:29.175392 10576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:41:29.175684 10576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [term 1 LEADER]: Becoming Leader. State: Replica: 66eeecb9e32f424a8fdd913b908b4a52, State: Running, Role: LEADER
I20260925 22:41:29.176038 10576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66eeecb9e32f424a8fdd913b908b4a52" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 41871 } }
I20260925 22:41:29.176718 10571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:41:29.177785 10578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66eeecb9e32f424a8fdd913b908b4a52. Latest consensus state: current_term: 1 leader_uuid: "66eeecb9e32f424a8fdd913b908b4a52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66eeecb9e32f424a8fdd913b908b4a52" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 41871 } } }
I20260925 22:41:29.178143 10577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66eeecb9e32f424a8fdd913b908b4a52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66eeecb9e32f424a8fdd913b908b4a52" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 41871 } } }
I20260925 22:41:29.178268 10577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:29.178498 10578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:29.178683 10585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:41:29.181533 10585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:41:29.187551 10585 catalog_manager.cc:1347] Generated new cluster ID: a9425dedb4b649fa8757d8816797ae91
I20260925 22:41:29.187642 10585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:41:29.198683 10585 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:41:29.200191 10585 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:41:29.208838 10585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52: Generated new TSK 0
I20260925 22:41:29.209829 10585 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
I20260925 22:41:29.245930 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:0
--local_ip_for_outbound_sockets=127.2.74.193
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:41871
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:37347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:29.364528 10597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:29.364880 10597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:29.364964 10597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:29.369282 10597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:29.369403 10597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:29.369504 10597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:41:29.375296 10597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:37347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:41871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:29.376540 10597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:29.377765 10597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:29.388182 10603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:29.388193 10602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:29.388265 10605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:29.388592 10597 server_base.cc:1034] running on GCE node
I20260925 22:41:29.389058 10597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:29.389851 10597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:29.391037 10597 hybrid_clock.cc:648] HybridClock initialized: now 1790376089391012 us; error 29 us; skew 500 ppm
Sep 25 22:41:29 dist-test-slave-csnl krb5kdc[10490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376089, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:29.394037 10597 init.cc:377] Logged in from keytab as kudu/127.2.74.193@KRBTEST.COM (short username kudu)
I20260925 22:41:29.395088 10597 webserver.cc:460] Webserver started at http://127.2.74.193:40969/ using document root <none> and password file <none>
I20260925 22:41:29.395709 10597 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:29.395809 10597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:29.396054 10597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:29.398309 10597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/data/instance:
uuid: "8dc3a36b504048daaa7b2767a4ce024a"
format_stamp: "Formatted at 2026-09-25 22:41:29 on dist-test-slave-csnl"
server_key: "2fc2f759afdc4574d4ec194ff8b20bba"
server_key_iv: "96dae6a5456e569cade5ea5f14355126"
server_key_version: "encryptionkey@0"
I20260925 22:41:29.398993 10597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance:
uuid: "8dc3a36b504048daaa7b2767a4ce024a"
format_stamp: "Formatted at 2026-09-25 22:41:29 on dist-test-slave-csnl"
server_key: "2fc2f759afdc4574d4ec194ff8b20bba"
server_key_iv: "96dae6a5456e569cade5ea5f14355126"
server_key_version: "encryptionkey@0"
I20260925 22:41:29.403362 10597 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:41:29.406201 10612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:29.407819 10597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:41:29.407998 10597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "8dc3a36b504048daaa7b2767a4ce024a"
format_stamp: "Formatted at 2026-09-25 22:41:29 on dist-test-slave-csnl"
server_key: "2fc2f759afdc4574d4ec194ff8b20bba"
server_key_iv: "96dae6a5456e569cade5ea5f14355126"
server_key_version: "encryptionkey@0"
I20260925 22:41:29.408152 10597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:29.422861 10597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:29.426524 10597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:29.426760 10597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:29.427496 10597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:29.428370 10597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:29.428445 10597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:29.428519 10597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:29.428562 10597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:29.437948 10597 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:37525
I20260925 22:41:29.437968 10725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:37525 every 8 connection(s)
I20260925 22:41:29.438917 10597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:41:29.443344 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 10597
I20260925 22:41:29.443487 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance
I20260925 22:41:29.443823 2347 external_mini_cluster.cc:1428] Setting key 05e8dd7385f66f5efec63365d2982190
Sep 25 22:41:29 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376089, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.74.194@KRBTEST.COM; defaulting to no policy
I20260925 22:41:29.456184 10574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:29.441023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:40495 (local address 127.2.74.254:41871)
0925 22:41:29.441185 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:29.441189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:29.442179 (+ 990us) server_negotiation.cc:407] Connection header received
0925 22:41:29.443977 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:29.443981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:29.444037 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:29.444123 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:29.445875 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:29.446403 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:29.447998 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:29.448439 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:29.448570 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:29.451759 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:29.451763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:29.451766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:29.451805 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:29.453600 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:29.454276 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:29.454283 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:29.454287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:29.454388 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:29.454747 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:29.454750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:29.454751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:29.455041 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:29.455174 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:29.455762 (+ 588us) server_negotiation.cc:299] Negotiation successful
0925 22:41:29.455957 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.194@KRBTEST.COM" created.
I20260925 22:41:29.457159 10728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:29.441474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.193:40495)
0925 22:41:29.441981 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:29.442033 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:29.443642 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:29.444363 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:29.444371 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:29.444933 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:29.445690 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:29.445712 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:29.446557 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:29.446563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:29.447824 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:29.447835 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:29.448619 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:29.448622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:29.448793 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:29.449503 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:29.449528 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:29.451499 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:29.453756 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:29.453763 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:29.453779 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:29.454148 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:29.454486 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:29.454488 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:29.454490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:29.454653 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:29.455174 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:29.455183 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:29.455479 (+ 296us) client_negotiation.cc:769] Sending connection context
0925 22:41:29.455701 (+ 222us) client_negotiation.cc:240] Negotiation successful
0925 22:41:29.456091 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":123,"threads_started":1}
I20260925 22:41:29.458645 10726 heartbeater.cc:347] Connected to a master server at 127.2.74.254:41871
I20260925 22:41:29.458930 10726 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:29.459543 10726 heartbeater.cc:511] Master 127.2.74.254:41871 requested a full tablet report, sending...
I20260925 22:41:29.461215 10535 ts_manager.cc:194] Registered new tserver with Master: 8dc3a36b504048daaa7b2767a4ce024a (127.2.74.193:37525)
I20260925 22:41:29.462402 10535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.193@KRBTEST.COM'} at 127.2.74.193:40495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
I20260925 22:41:29.474200 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:0
--local_ip_for_outbound_sockets=127.2.74.194
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:41871
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:37347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:29.592967 10731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:29.593274 10731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:29.593322 10731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:29.597710 10731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:29.597797 10731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:29.597860 10731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:41:29.603853 10731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:37347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:41871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:29.605017 10731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:29.606110 10731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:41:29.616822 10731 server_base.cc:1034] running on GCE node
W20260925 22:41:29.616779 10736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:29.616801 10737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:29.616858 10739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:29.617532 10731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:29.618186 10731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:29.619503 10731 hybrid_clock.cc:648] HybridClock initialized: now 1790376089619455 us; error 71 us; skew 500 ppm
Sep 25 22:41:29 dist-test-slave-csnl krb5kdc[10490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376089, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:29.622473 10731 init.cc:377] Logged in from keytab as kudu/127.2.74.194@KRBTEST.COM (short username kudu)
I20260925 22:41:29.623565 10731 webserver.cc:460] Webserver started at http://127.2.74.194:33027/ using document root <none> and password file <none>
I20260925 22:41:29.624171 10731 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:29.624266 10731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:29.624492 10731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:29.626667 10731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/data/instance:
uuid: "83a78ee2e9b242449facfb7399173b97"
format_stamp: "Formatted at 2026-09-25 22:41:29 on dist-test-slave-csnl"
server_key: "7db26c57d35822bc3a0c5e937de98b12"
server_key_iv: "307a24bd51636a3f52c655f234bb5fc8"
server_key_version: "encryptionkey@0"
I20260925 22:41:29.627316 10731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance:
uuid: "83a78ee2e9b242449facfb7399173b97"
format_stamp: "Formatted at 2026-09-25 22:41:29 on dist-test-slave-csnl"
server_key: "7db26c57d35822bc3a0c5e937de98b12"
server_key_iv: "307a24bd51636a3f52c655f234bb5fc8"
server_key_version: "encryptionkey@0"
I20260925 22:41:29.632648 10731 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 22:41:29.635372 10746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:29.636811 10731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:41:29.636988 10731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "83a78ee2e9b242449facfb7399173b97"
format_stamp: "Formatted at 2026-09-25 22:41:29 on dist-test-slave-csnl"
server_key: "7db26c57d35822bc3a0c5e937de98b12"
server_key_iv: "307a24bd51636a3f52c655f234bb5fc8"
server_key_version: "encryptionkey@0"
I20260925 22:41:29.637133 10731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:29.662566 10731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:29.666194 10731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:29.666411 10731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:29.667119 10731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:29.667977 10731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:29.668049 10731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:29.668114 10731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:29.668165 10731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:29.677855 10731 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:37637
I20260925 22:41:29.677884 10859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:37637 every 8 connection(s)
I20260925 22:41:29.679088 10731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:41:29.682509 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 10731
I20260925 22:41:29.682667 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance
I20260925 22:41:29.683043 2347 external_mini_cluster.cc:1428] Setting key 5798467df9720896102674b957c3a138
Sep 25 22:41:29 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376089, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.74.195@KRBTEST.COM; defaulting to no policy
I20260925 22:41:29.695381 10574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:29.681035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:49555 (local address 127.2.74.254:41871)
0925 22:41:29.681175 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:29.681179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:29.682012 (+ 833us) server_negotiation.cc:407] Connection header received
0925 22:41:29.683755 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:29.683758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:29.683847 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:29.683934 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:29.685663 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:29.686277 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:29.687663 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:29.687910 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:29.688050 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:29.691203 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:29.691206 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:29.691209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:29.691236 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:29.693030 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:29.693677 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:29.693681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:29.693682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:29.693771 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:29.694221 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:29.694224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:29.694225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:29.694412 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:29.694488 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:29.695107 (+ 619us) server_negotiation.cc:299] Negotiation successful
0925 22:41:29.695256 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.195@KRBTEST.COM" created.
I20260925 22:41:29.696432 10862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:29.681343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.194:49555)
0925 22:41:29.681836 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:29.681887 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:29.683465 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:29.684108 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:29.684117 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:29.684677 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:29.685463 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:29.685499 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:29.686424 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:29.686427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:29.687524 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:29.687534 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:29.688103 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:29.688106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:29.688290 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:29.688967 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:29.688986 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:29.690987 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:29.693174 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:29.693180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:29.693196 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:29.693554 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:29.693904 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:29.693907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:29.693909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:29.694083 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:29.694504 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:29.694509 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:29.694784 (+ 275us) client_negotiation.cc:769] Sending connection context
0925 22:41:29.695008 (+ 224us) client_negotiation.cc:240] Negotiation successful
0925 22:41:29.695354 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":113,"threads_started":1}
I20260925 22:41:29.697755 10860 heartbeater.cc:347] Connected to a master server at 127.2.74.254:41871
I20260925 22:41:29.698024 10860 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:29.698552 10860 heartbeater.cc:511] Master 127.2.74.254:41871 requested a full tablet report, sending...
I20260925 22:41:29.699790 10535 ts_manager.cc:194] Registered new tserver with Master: 83a78ee2e9b242449facfb7399173b97 (127.2.74.194:37637)
I20260925 22:41:29.700692 10535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.194@KRBTEST.COM'} at 127.2.74.194:49555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
I20260925 22:41:29.715374 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:0
--local_ip_for_outbound_sockets=127.2.74.195
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:41871
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:37347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:29.830631 10865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:29.830931 10865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:29.831019 10865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:29.835280 10865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:29.835357 10865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:29.835419 10865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:41:29.841061 10865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:37347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:41871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:29.842295 10865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:29.843348 10865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:29.853852 10873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:29.853833 10871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:29.853849 10870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:29.854038 10865 server_base.cc:1034] running on GCE node
I20260925 22:41:29.854736 10865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:29.855516 10865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:29.856693 10865 hybrid_clock.cc:648] HybridClock initialized: now 1790376089856654 us; error 55 us; skew 500 ppm
Sep 25 22:41:29 dist-test-slave-csnl krb5kdc[10490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376089, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:29.859826 10865 init.cc:377] Logged in from keytab as kudu/127.2.74.195@KRBTEST.COM (short username kudu)
I20260925 22:41:29.860836 10865 webserver.cc:460] Webserver started at http://127.2.74.195:46079/ using document root <none> and password file <none>
I20260925 22:41:29.861491 10865 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:29.861593 10865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:29.861846 10865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:29.864131 10865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/data/instance:
uuid: "3151ce9ccda64ae0b301d7797751b891"
format_stamp: "Formatted at 2026-09-25 22:41:29 on dist-test-slave-csnl"
server_key: "6540fe85ff6ec772cc37201526fc2794"
server_key_iv: "5d2dffc8893bc70ce237d7c169765117"
server_key_version: "encryptionkey@0"
I20260925 22:41:29.864768 10865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance:
uuid: "3151ce9ccda64ae0b301d7797751b891"
format_stamp: "Formatted at 2026-09-25 22:41:29 on dist-test-slave-csnl"
server_key: "6540fe85ff6ec772cc37201526fc2794"
server_key_iv: "5d2dffc8893bc70ce237d7c169765117"
server_key_version: "encryptionkey@0"
I20260925 22:41:29.868970 10865 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:41:29.871879 10880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:29.873256 10865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:41:29.873432 10865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "3151ce9ccda64ae0b301d7797751b891"
format_stamp: "Formatted at 2026-09-25 22:41:29 on dist-test-slave-csnl"
server_key: "6540fe85ff6ec772cc37201526fc2794"
server_key_iv: "5d2dffc8893bc70ce237d7c169765117"
server_key_version: "encryptionkey@0"
I20260925 22:41:29.873579 10865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:29.888581 10865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:29.892202 10865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:29.892427 10865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:29.893064 10865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:29.893927 10865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:29.894001 10865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:29.894073 10865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:29.894116 10865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:29.903512 10865 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:37549
I20260925 22:41:29.903554 10993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:37549 every 8 connection(s)
I20260925 22:41:29.904477 10865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:41:29.912062 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 10865
I20260925 22:41:29.912217 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance
I20260925 22:41:29.912540 2347 external_mini_cluster.cc:1428] Setting key 4f6ad4afd544ed58e61d0a3f0cd60dbe
Sep 25 22:41:29 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376089, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:41:29.918740 10574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:29.906420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:55101 (local address 127.2.74.254:41871)
0925 22:41:29.906562 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:29.906566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:29.907608 (+ 1042us) server_negotiation.cc:407] Connection header received
0925 22:41:29.908928 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:29.908931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:29.908981 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:29.909059 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:29.910313 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:29.910840 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:29.912319 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:29.912541 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:29.912696 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:29.915478 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:29.915481 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:29.915483 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:29.915508 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:29.916748 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:29.917221 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:29.917224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:29.917225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:29.917295 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:29.917584 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:29.917586 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:29.917587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:29.917815 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:29.917893 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:29.918446 (+ 553us) server_negotiation.cc:299] Negotiation successful
0925 22:41:29.918578 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:41:29.919697 10996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:29.906847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.195:55101)
0925 22:41:29.907414 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:29.907473 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:29.908720 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:29.909243 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:29.909249 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:29.909652 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:29.910154 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:29.910168 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:29.910983 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:29.910986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:29.912116 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:29.912131 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:29.912748 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:29.912752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:29.912958 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:29.913595 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:29.913613 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:29.915311 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:29.916851 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:29.916856 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:29.916870 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:29.917127 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:29.917367 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:29.917369 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:29.917371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:29.917497 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:29.917941 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:29.917946 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:29.918211 (+ 265us) client_negotiation.cc:769] Sending connection context
0925 22:41:29.918388 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 22:41:29.918706 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":140,"threads_started":1}
I20260925 22:41:29.921000 10994 heartbeater.cc:347] Connected to a master server at 127.2.74.254:41871
I20260925 22:41:29.921253 10994 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:29.921792 10994 heartbeater.cc:511] Master 127.2.74.254:41871 requested a full tablet report, sending...
I20260925 22:41:29.922925 10535 ts_manager.cc:194] Registered new tserver with Master: 3151ce9ccda64ae0b301d7797751b891 (127.2.74.195:37549)
I20260925 22:41:29.923583 10535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.195@KRBTEST.COM'} at 127.2.74.195:55101
I20260925 22:41:29.926613 2347 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.193.keytab.
Entry for principal oryx/127.2.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.193.keytab.
WARNING: no policy specified for oryx/127.2.74.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.74.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.194.keytab.
Entry for principal oryx/127.2.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.194.keytab.
WARNING: no policy specified for oryx/127.2.74.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.74.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.195.keytab.
Entry for principal oryx/127.2.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.195.keytab.
I20260925 22:41:30.001750 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 10597
I20260925 22:41:30.007691 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 10731
I20260925 22:41:30.013365 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 10865
I20260925 22:41:30.019181 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 10503
I20260925 22:41:30.025550 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:41871
--webserver_interface=127.2.74.254
--webserver_port=45823
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:37347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:41871
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:30.144877 11003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:30.145191 11003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:30.145236 11003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:30.149672 11003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:41:30.149756 11003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:30.149776 11003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:30.149794 11003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:41:30.149840 11003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:41:30.155740 11003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:37347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:41871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:41871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=45823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:30.156987 11003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:30.158118 11003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:30.167578 11009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:30.167603 11008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:30.167625 11011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:30.167603 11003 server_base.cc:1034] running on GCE node
I20260925 22:41:30.168527 11003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:30.169884 11003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:30.171124 11003 hybrid_clock.cc:648] HybridClock initialized: now 1790376090171095 us; error 62 us; skew 500 ppm
Sep 25 22:41:30 dist-test-slave-csnl krb5kdc[10490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376090, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:30.174284 11003 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:41:30.175387 11003 webserver.cc:460] Webserver started at http://127.2.74.254:45823/ using document root <none> and password file <none>
I20260925 22:41:30.175992 11003 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:30.176106 11003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:30.180148 11003 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260925 22:41:30.182524 11018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:30.183972 11003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:41:30.184139 11003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "66eeecb9e32f424a8fdd913b908b4a52"
format_stamp: "Formatted at 2026-09-25 22:41:29 on dist-test-slave-csnl"
server_key: "1ef2ddd5c70992c1beacbe0fd5b76f0b"
server_key_iv: "5a9eaac066befa58b72fabf2284bfc55"
server_key_version: "encryptionkey@0"
I20260925 22:41:30.184723 11003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:30.204617 11003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:30.208165 11003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:30.208402 11003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:30.216092 11003 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:41871
I20260925 22:41:30.216120 11070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:41871 every 8 connection(s)
I20260925 22:41:30.217263 11003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:41:30.221937 11071 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:41:30.222421 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 11003
I20260925 22:41:30.222889 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:37525
--local_ip_for_outbound_sockets=127.2.74.193
--tserver_master_addrs=127.2.74.254:41871
--webserver_port=40969
--webserver_interface=127.2.74.193
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:37347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.74.193
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:41:30.224068 11071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52: Bootstrap starting.
I20260925 22:41:30.247900 11071 log.cc:824] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:30.256652 11071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:41:30.257141 11071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52: Bootstrap complete.
I20260925 22:41:30.262835 11071 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66eeecb9e32f424a8fdd913b908b4a52" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 41871 } }
I20260925 22:41:30.263405 11071 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66eeecb9e32f424a8fdd913b908b4a52, State: Initialized, Role: FOLLOWER
I20260925 22:41:30.263983 11071 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66eeecb9e32f424a8fdd913b908b4a52" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 41871 } }
I20260925 22:41:30.264130 11071 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:41:30.264196 11071 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:41:30.264292 11071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:41:30.266989 11071 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66eeecb9e32f424a8fdd913b908b4a52" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 41871 } }
I20260925 22:41:30.267419 11071 leader_election.cc:302] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66eeecb9e32f424a8fdd913b908b4a52; no voters:
I20260925 22:41:30.267836 11071 leader_election.cc:288] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 22:41:30.268003 11075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 22:41:30.268293 11075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [term 2 LEADER]: Becoming Leader. State: Replica: 66eeecb9e32f424a8fdd913b908b4a52, State: Running, Role: LEADER
I20260925 22:41:30.268687 11075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66eeecb9e32f424a8fdd913b908b4a52" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 41871 } }
I20260925 22:41:30.269191 11071 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:41:30.270115 11076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "66eeecb9e32f424a8fdd913b908b4a52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66eeecb9e32f424a8fdd913b908b4a52" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 41871 } } }
I20260925 22:41:30.270241 11076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:30.271013 11077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66eeecb9e32f424a8fdd913b908b4a52. Latest consensus state: current_term: 2 leader_uuid: "66eeecb9e32f424a8fdd913b908b4a52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66eeecb9e32f424a8fdd913b908b4a52" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 41871 } } }
I20260925 22:41:30.271108 11077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52 [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:30.271145 11084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:41:30.274179 11084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:41:30.274783 11084 catalog_manager.cc:1259] Loaded cluster ID: a9425dedb4b649fa8757d8816797ae91
I20260925 22:41:30.274868 11084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:41:30.276608 11084 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:41:30.277252 11084 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 66eeecb9e32f424a8fdd913b908b4a52: Loaded TSK: 0
I20260925 22:41:30.278368 11084 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:41:30.366933 11073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:30.367311 11073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:30.367357 11073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:30.372104 11073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:30.372193 11073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:30.372259 11073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:41:30.377981 11073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:37347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.193.keytab
--principal=oryx/127.2.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:37525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=40969
--tserver_master_addrs=127.2.74.254:41871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:30.379215 11073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:30.380343 11073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:30.391490 11098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:30.391546 11101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:30.391490 11099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:30.391853 11073 server_base.cc:1034] running on GCE node
I20260925 22:41:30.392398 11073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:30.393155 11073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:30.394344 11073 hybrid_clock.cc:648] HybridClock initialized: now 1790376090394314 us; error 60 us; skew 500 ppm
Sep 25 22:41:30 dist-test-slave-csnl krb5kdc[10490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376090, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:30.397465 11073 init.cc:377] Logged in from keytab as oryx/127.2.74.193@KRBTEST.COM (short username oryx)
I20260925 22:41:30.398423 11073 webserver.cc:460] Webserver started at http://127.2.74.193:40969/ using document root <none> and password file <none>
I20260925 22:41:30.399101 11073 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:30.399225 11073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:30.403644 11073 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:41:30.405977 11108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:30.407397 11073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:41:30.407543 11073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "8dc3a36b504048daaa7b2767a4ce024a"
format_stamp: "Formatted at 2026-09-25 22:41:29 on dist-test-slave-csnl"
server_key: "2fc2f759afdc4574d4ec194ff8b20bba"
server_key_iv: "96dae6a5456e569cade5ea5f14355126"
server_key_version: "encryptionkey@0"
I20260925 22:41:30.408130 11073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:30.425027 11073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:30.428596 11073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:30.428818 11073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:30.429430 11073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:30.430276 11073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:30.430346 11073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:30.430419 11073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:30.430472 11073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:30.439715 11073 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:37525
I20260925 22:41:30.439738 11221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:37525 every 8 connection(s)
I20260925 22:41:30.440831 11073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:41:30.451157 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 11073
I20260925 22:41:30.451663 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:37637
--local_ip_for_outbound_sockets=127.2.74.194
--tserver_master_addrs=127.2.74.254:41871
--webserver_port=33027
--webserver_interface=127.2.74.194
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:37347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.74.194
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:41:30 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:30 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 22:41:30.454825 11224 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:30.443058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:33539 (local address 127.2.74.254:41871)
0925 22:41:30.443581 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:30.443606 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:30.443824 (+ 218us) server_negotiation.cc:407] Connection header received
0925 22:41:30.445381 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:30.445392 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:30.445716 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:30.446054 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:30.447327 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.448321 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.449525 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.450017 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.450204 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:30.454149 (+ 3945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.193:33539: BlockingRecv error: recv got EOF from 127.2.74.193:33539 (error 108)
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":144,"threads_started":1}
I20260925 22:41:30.454815 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:30.443178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.193:33539)
0925 22:41:30.443640 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:30.443698 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:30.444939 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:30.446132 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:30.446140 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:30.446581 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:30.447138 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:30.447155 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.448485 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.448488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:30.449384 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:30.449395 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.450203 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.450208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:30.450400 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:30.451072 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:30.451091 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:30.453796 (+ 2705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":132,"threads_started":1}
W20260925 22:41:30.455590 11222 heartbeater.cc:644] Failed to heartbeat to 127.2.74.254:41871 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.74.254:41871: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Sep 25 22:41:30 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:30 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:30.462411 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:30.456119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.193:48999)
0925 22:41:30.456316 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:30.456335 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:30.456451 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:30.456720 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:30.456725 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:30.456938 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:30.457113 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:30.457122 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.457853 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.457860 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:30.458936 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:30.459010 (+ 74us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.459459 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.459465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:30.459594 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:30.460157 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:30.460179 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:30.462297 (+ 2118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":30}
W20260925 22:41:30.462689 11224 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:30.456193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:48999 (local address 127.2.74.254:41871)
0925 22:41:30.456307 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:30.456313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:30.456404 (+ 91us) server_negotiation.cc:407] Connection header received
0925 22:41:30.456548 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:30.456551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:30.456596 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:30.456666 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:30.457229 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.457735 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.459119 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.459336 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.459415 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:30.462520 (+ 3105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.193:48999: BlockingRecv error: recv got EOF from 127.2.74.193:48999 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 22:41:30 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:30 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:30.468739 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:30.463170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.193:38747)
0925 22:41:30.463272 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:30.463288 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:30.463369 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:30.463698 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:30.463701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:30.463843 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:30.463927 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:30.463932 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.465032 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.465035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:30.465590 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:30.465618 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.466178 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.466181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:30.466242 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:30.466793 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:30.466805 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:30.468629 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 22:41:30.468905 11224 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:30.463257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:38747 (local address 127.2.74.254:41871)
0925 22:41:30.463399 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:30.463405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:30.463424 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:41:30.463478 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:30.463484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:30.463543 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:30.463646 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:30.464058 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.464873 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.465722 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.466071 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.466168 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:30.468784 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.193:38747: BlockingRecv error: recv got EOF from 127.2.74.193:38747 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 22:41:30.469164 11222 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:41:30.574750 11226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:30.575088 11226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:30.575135 11226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:30.579428 11226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:30.579506 11226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:30.579571 11226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:41:30.585237 11226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:37347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.194.keytab
--principal=oryx/127.2.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:37637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=33027
--tserver_master_addrs=127.2.74.254:41871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:30.586452 11226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:30.587630 11226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:30.598366 11231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:30.598376 11232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:30.598381 11234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:30.598928 11226 server_base.cc:1034] running on GCE node
I20260925 22:41:30.599406 11226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:30.600107 11226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:30.601289 11226 hybrid_clock.cc:648] HybridClock initialized: now 1790376090601241 us; error 66 us; skew 500 ppm
Sep 25 22:41:30 dist-test-slave-csnl krb5kdc[10490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376090, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:30.604573 11226 init.cc:377] Logged in from keytab as oryx/127.2.74.194@KRBTEST.COM (short username oryx)
I20260925 22:41:30.605619 11226 webserver.cc:460] Webserver started at http://127.2.74.194:33027/ using document root <none> and password file <none>
I20260925 22:41:30.606208 11226 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:30.606292 11226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:30.610529 11226 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:41:30.612987 11241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:30.614377 11226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:41:30.614523 11226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "83a78ee2e9b242449facfb7399173b97"
format_stamp: "Formatted at 2026-09-25 22:41:29 on dist-test-slave-csnl"
server_key: "7db26c57d35822bc3a0c5e937de98b12"
server_key_iv: "307a24bd51636a3f52c655f234bb5fc8"
server_key_version: "encryptionkey@0"
I20260925 22:41:30.615182 11226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:30.640659 11226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:30.644516 11226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:30.644744 11226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:30.645395 11226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:30.646296 11226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:30.646343 11226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:30.646416 11226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:30.646461 11226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:30.655877 11226 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:37637
I20260925 22:41:30.655916 11354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:37637 every 8 connection(s)
I20260925 22:41:30.656955 11226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:41:30.659590 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 11226
I20260925 22:41:30.660081 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:37549
--local_ip_for_outbound_sockets=127.2.74.195
--tserver_master_addrs=127.2.74.254:41871
--webserver_port=46079
--webserver_interface=127.2.74.195
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:37347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.74.195
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:41:30 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:30 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 22:41:30.671164 11224 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:30.658936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:50435 (local address 127.2.74.254:41871)
0925 22:41:30.659143 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:30.659147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:30.660107 (+ 960us) server_negotiation.cc:407] Connection header received
0925 22:41:30.661636 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:30.661640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:30.661698 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:30.661783 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:30.663631 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.664243 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.666348 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.666573 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.666665 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:30.671024 (+ 4359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.194:50435: BlockingRecv error: recv got EOF from 127.2.74.194:50435 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 22:41:30.671844 11357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:30.659416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.194:50435)
0925 22:41:30.659900 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:30.659968 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:30.661322 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:30.662015 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:30.662029 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:30.662611 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:30.663429 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:30.663455 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.664417 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.664424 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:30.666175 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:30.666192 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.666768 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.666775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:30.667107 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:30.667824 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:30.667853 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:30.670873 (+ 3020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":131,"threads_started":1}
W20260925 22:41:30.672348 11355 heartbeater.cc:644] Failed to heartbeat to 127.2.74.254:41871 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.74.254:41871: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Sep 25 22:41:30 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:30 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:30.678367 11357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:30.672834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.194:56751)
0925 22:41:30.673065 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:30.673086 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:30.673210 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:30.673468 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:30.673471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:30.673640 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:30.673733 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:30.673738 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.674494 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.674500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:30.675206 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:30.675220 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.675829 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.675832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:30.675903 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:30.676436 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:30.676450 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:30.678257 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 22:41:30.678692 11224 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:30.672931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:56751 (local address 127.2.74.254:41871)
0925 22:41:30.673031 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:30.673035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:30.673154 (+ 119us) server_negotiation.cc:407] Connection header received
0925 22:41:30.673311 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:30.673314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:30.673365 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:30.673452 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:30.673852 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.674385 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.675346 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.675714 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.675822 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:30.678437 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.194:56751: BlockingRecv error: recv got EOF from 127.2.74.194:56751 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 25 22:41:30 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:30 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:30.684967 11357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:30.679320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.194:54953)
0925 22:41:30.679444 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:30.679460 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:30.679543 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:30.679866 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:30.679870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:30.680036 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:30.680127 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:30.680132 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.681174 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.681176 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:30.681721 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:30.681729 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.682305 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.682308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:30.682371 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:30.682873 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:30.682885 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:30.684852 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 22:41:30.685151 11224 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:30.679388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:54953 (local address 127.2.74.254:41871)
0925 22:41:30.679516 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:30.679523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:30.679542 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:41:30.679672 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:30.679675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:30.679723 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:30.679805 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:30.680234 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.681060 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.681833 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.682191 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.682293 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:30.685032 (+ 2739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.194:54953: BlockingRecv error: recv got EOF from 127.2.74.194:54953 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 22:41:30.685405 11355 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:41:30.783984 11358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:30.784284 11358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:30.784330 11358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:30.788941 11358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:30.789047 11358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:30.789181 11358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:41:30.795054 11358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:37347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/krb5kdc/oryx_127.2.74.195.keytab
--principal=oryx/127.2.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:37549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=46079
--tserver_master_addrs=127.2.74.254:41871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:30.796192 11358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:30.797286 11358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:30.808004 11363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:30.808058 11366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:30.808059 11364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:30.808281 11358 server_base.cc:1034] running on GCE node
I20260925 22:41:30.808772 11358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:30.809530 11358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:30.810698 11358 hybrid_clock.cc:648] HybridClock initialized: now 1790376090810682 us; error 31 us; skew 500 ppm
Sep 25 22:41:30 dist-test-slave-csnl krb5kdc[10490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376090, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:30.813712 11358 init.cc:377] Logged in from keytab as oryx/127.2.74.195@KRBTEST.COM (short username oryx)
I20260925 22:41:30.814682 11358 webserver.cc:460] Webserver started at http://127.2.74.195:46079/ using document root <none> and password file <none>
I20260925 22:41:30.815363 11358 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:30.815451 11358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:30.819990 11358 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:41:30.822337 11373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:30.823834 11358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:41:30.824003 11358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "3151ce9ccda64ae0b301d7797751b891"
format_stamp: "Formatted at 2026-09-25 22:41:29 on dist-test-slave-csnl"
server_key: "6540fe85ff6ec772cc37201526fc2794"
server_key_iv: "5d2dffc8893bc70ce237d7c169765117"
server_key_version: "encryptionkey@0"
I20260925 22:41:30.824611 11358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:30.851481 11358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:30.854653 11358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:30.854874 11358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:30.855541 11358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:30.856361 11358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:30.856408 11358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:30.856448 11358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:30.856464 11358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:30.865554 11358 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:37549
I20260925 22:41:30.865577 11486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:37549 every 8 connection(s)
I20260925 22:41:30.866626 11358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:41:30.867892 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 11358
Sep 25 22:41:30 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:30 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 22:41:30.878132 11490 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:30.869036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:49553 (local address 127.2.74.254:41871)
0925 22:41:30.869648 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:30.869653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:30.870363 (+ 710us) server_negotiation.cc:407] Connection header received
0925 22:41:30.871358 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:30.871362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:30.871411 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:30.871495 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:30.872664 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.873185 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.874343 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.874586 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.874705 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:30.877996 (+ 3291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.195:49553: BlockingRecv error: recv got EOF from 127.2.74.195:49553 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":157,"threads_started":1}
I20260925 22:41:30.878275 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:30.868654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43590 (local address 127.2.74.254:41871)
0925 22:41:30.868796 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:30.868800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:30.869140 (+ 340us) server_negotiation.cc:407] Connection header received
0925 22:41:30.869309 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:30.869314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:30.869389 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:30.869491 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:30.870380 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.870921 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.871913 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.872199 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.872306 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:30.873794 (+ 1488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:30.873803 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:30.873824 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:30.873852 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:30.875828 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:30.876316 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:30.876329 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:30.876335 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:30.876494 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:30.876807 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:30.876811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:30.876813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:30.877248 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:30.877455 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:30.877793 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 22:41:30.878054 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 22:41:30.878923 11491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:30.869218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.195:49553)
0925 22:41:30.869989 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:30.870041 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:30.871121 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:30.871675 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:30.871683 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:30.872029 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:30.872515 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:30.872528 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.873316 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.873320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:30.874213 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:30.874222 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.874742 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.874746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:30.874939 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:30.875505 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:30.875521 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:30.877880 (+ 2359us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":131,"threads_started":1}
W20260925 22:41:30.879753 11487 heartbeater.cc:644] Failed to heartbeat to 127.2.74.254:41871 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.74.254:41871: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Sep 25 22:41:30 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:30 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:30.885509 11491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:30.880256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.195:48683)
0925 22:41:30.880361 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:30.880374 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:30.880440 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:30.880783 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:30.880786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:30.880982 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:30.881095 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:30.881102 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.881947 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.881949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:30.882533 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:30.882541 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.883005 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.883010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:30.883089 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:30.883564 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:30.883576 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:30.885395 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 22:41:30.885694 11224 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:30.880476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:48683 (local address 127.2.74.254:41871)
0925 22:41:30.880598 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:30.880601 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:30.880611 (+ 10us) server_negotiation.cc:407] Connection header received
0925 22:41:30.880638 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:30.880640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:30.880678 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:30.880767 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:30.881209 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.881839 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.882641 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.882855 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.882933 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:30.885553 (+ 2620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.195:48683: BlockingRecv error: recv got EOF from 127.2.74.195:48683 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 22:41:30 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:30 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:30.891414 11491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:30.886129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.195:52069)
0925 22:41:30.886248 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:30.886264 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:30.886343 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:30.886543 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:30.886545 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:30.886684 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:30.886764 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:30.886768 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.887536 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.887538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:30.888071 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:30.888079 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.888658 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.888661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:30.888726 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:30.889250 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:30.889272 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:30.891301 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 22:41:30.891580 11224 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:30.886216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:52069 (local address 127.2.74.254:41871)
0925 22:41:30.886355 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:30.886358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:30.886371 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:41:30.886409 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:30.886411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:30.886445 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:30.886514 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:30.886862 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.887439 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.888204 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:30.888554 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:30.888635 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:30.891464 (+ 2829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.195:52069: BlockingRecv error: recv got EOF from 127.2.74.195:52069 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 22:41:30.891773 11487 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 22:41:31 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:31 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:31.475968 11492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:31.469915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.193:53759)
0925 22:41:31.470230 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:31.470249 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:31.470369 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:31.470767 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:31.470773 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:31.471046 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:31.471232 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:31.471241 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:31.471995 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:31.471999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:31.472581 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:31.472588 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:31.473064 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:31.473067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:31.473144 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:31.473764 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:31.473786 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:31.475842 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":103,"threads_started":1}
W20260925 22:41:31.476203 11493 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:31.469986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:53759 (local address 127.2.74.254:41871)
0925 22:41:31.470337 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:31.470342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:31.470359 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:41:31.470483 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:31.470487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:31.470554 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:31.470648 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:31.471370 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:31.471896 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:31.472711 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:31.472946 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:31.473031 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:31.476040 (+ 3009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.193:53759: BlockingRecv error: recv got EOF from 127.2.74.193:53759 (error 108)
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":59,"threads_started":1}
Sep 25 22:41:31 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:31 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:31.692236 11494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:31.686254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.194:53051)
0925 22:41:31.686549 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:31.686563 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:31.686684 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:31.687061 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:31.687064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:31.687287 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:31.687388 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:31.687394 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:31.688204 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:31.688210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:31.688866 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:31.688875 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:31.689333 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:31.689337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:31.689418 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:31.690025 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:31.690045 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:31.692064 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":82,"threads_started":1}
W20260925 22:41:31.692339 11493 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:31.686365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:53051 (local address 127.2.74.254:41871)
0925 22:41:31.686588 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:31.686592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:31.686611 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:41:31.686800 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:31.686804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:31.686861 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:31.687008 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:31.687519 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:31.688069 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:31.688994 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:31.689220 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:31.689331 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:31.692225 (+ 2894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.194:53051: BlockingRecv error: recv got EOF from 127.2.74.194:53051 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 25 22:41:31 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:31 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:31.898128 11495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:31.892469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.195:60899)
0925 22:41:31.892754 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:31.892768 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:31.892922 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:31.893256 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:31.893260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:31.893470 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:31.893569 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:31.893575 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:31.894288 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:31.894290 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:31.894878 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:31.894886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:31.895430 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:31.895433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:31.895505 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:31.896143 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:31.896156 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:31.898000 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":79,"threads_started":1}
W20260925 22:41:31.898325 11493 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:31.892542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:60899 (local address 127.2.74.254:41871)
0925 22:41:31.892703 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:31.892707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:31.892853 (+ 146us) server_negotiation.cc:407] Connection header received
0925 22:41:31.893019 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:31.893021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:31.893068 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:31.893181 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:31.893681 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:31.894192 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:31.895049 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:31.895272 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:31.895356 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:31.898160 (+ 2804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.195:60899: BlockingRecv error: recv got EOF from 127.2.74.195:60899 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 22:41:32 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:32 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:32.482273 11496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:32.476860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.193:56641)
0925 22:41:32.477179 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:32.477194 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:32.477290 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:32.477545 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:32.477547 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:32.477732 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:32.477828 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:32.477833 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:32.478553 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:32.478556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:32.479139 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:32.479146 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:32.479567 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:32.479569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:32.479627 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:32.480232 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:32.480244 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:32.482137 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":109,"threads_started":1}
W20260925 22:41:32.482431 11497 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:32.477068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:56641 (local address 127.2.74.254:41871)
0925 22:41:32.477328 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:32.477332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:32.477346 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:41:32.477386 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:32.477389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:32.477430 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:32.477509 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:32.477937 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:32.478456 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:32.479244 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:32.479467 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:32.479546 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:32.482290 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.193:56641: BlockingRecv error: recv got EOF from 127.2.74.193:56641 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":105,"threads_started":1}
Sep 25 22:41:32 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:32 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:32.699913 11498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:32.693837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.194:34475)
0925 22:41:32.694175 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:32.694187 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:32.694340 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:32.694636 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:32.694639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:32.694864 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:32.694995 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:32.695002 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:32.695963 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:32.695966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:32.696827 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:32.696838 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:32.697277 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:32.697279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:32.697342 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:32.697847 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:32.697860 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:32.699717 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":89,"threads_started":1}
W20260925 22:41:32.699961 11497 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:32.693926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:34475 (local address 127.2.74.254:41871)
0925 22:41:32.694113 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:32.694118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:32.694278 (+ 160us) server_negotiation.cc:407] Connection header received
0925 22:41:32.694460 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:32.694464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:32.694517 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:32.694613 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:32.695151 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:32.695778 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:32.696938 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:32.697160 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:32.697254 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:32.699855 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.194:34475: BlockingRecv error: recv got EOF from 127.2.74.194:34475 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 25 22:41:32 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:32 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:32.905305 11499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:32.899113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.195:58039)
0925 22:41:32.899470 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:32.899510 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:32.899635 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:32.900437 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:32.900442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:32.900704 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:32.900827 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:32.900835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:32.901605 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:32.901608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:32.902164 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:32.902171 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:32.902663 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:32.902665 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:32.902731 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:32.903336 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:32.903348 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:32.905147 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":125,"threads_started":1}
W20260925 22:41:32.905422 11497 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:32.899272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:58039 (local address 127.2.74.254:41871)
0925 22:41:32.899422 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:32.899426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:32.899590 (+ 164us) server_negotiation.cc:407] Connection header received
0925 22:41:32.900230 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:32.900233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:32.900287 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:32.900376 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:32.900964 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:32.901483 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:32.902317 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:32.902542 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:32.902646 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:32.905279 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.195:58039: BlockingRecv error: recv got EOF from 127.2.74.195:58039 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 22:41:33 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:33 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:33.494585 11500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:33.483248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.193:58273)
0925 22:41:33.483609 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:33.483623 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:33.483731 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:33.484000 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:33.484002 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:33.484183 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:33.484282 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:33.484288 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:33.485015 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:33.485018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:33.485606 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:33.485614 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:33.486200 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:33.486204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:33.486298 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:33.492120 (+ 5822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:33.492146 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:33.494383 (+ 2237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":139,"threads_started":1}
W20260925 22:41:33.494678 11501 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:33.483396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:58273 (local address 127.2.74.254:41871)
0925 22:41:33.483638 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:33.483642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:33.483680 (+ 38us) server_negotiation.cc:407] Connection header received
0925 22:41:33.483821 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:33.483824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:33.483868 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:33.483957 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:33.484388 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:33.484910 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:33.485735 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:33.486027 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:33.486169 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:33.494539 (+ 8370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.193:58273: BlockingRecv error: recv got EOF from 127.2.74.193:58273 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":102,"threads_started":1}
Sep 25 22:41:33 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:33 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:33.706619 11502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:33.700786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.194:34833)
0925 22:41:33.701114 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:33.701130 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:33.701289 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:33.701589 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:33.701592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:33.701817 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:33.701954 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:33.701962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:33.702841 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:33.702843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:33.703470 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:33.703479 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:33.703903 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:33.703905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:33.703968 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:33.704544 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:33.704556 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:33.706438 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":86,"threads_started":1}
W20260925 22:41:33.706677 11501 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:33.700935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:34833 (local address 127.2.74.254:41871)
0925 22:41:33.701114 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:33.701120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:33.701220 (+ 100us) server_negotiation.cc:407] Connection header received
0925 22:41:33.701414 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:33.701416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:33.701463 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:33.701567 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:33.702103 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:33.702742 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:33.703579 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:33.703806 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:33.703884 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:33.706571 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.194:34833: BlockingRecv error: recv got EOF from 127.2.74.194:34833 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 22:41:33 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:33 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:33.911857 11503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:33.906111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.195:43057)
0925 22:41:33.906417 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:33.906430 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:33.906564 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:33.906852 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:33.906855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:33.907094 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:33.907187 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:33.907193 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:33.907951 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:33.907956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:33.908521 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:33.908530 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:33.908988 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:33.908991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:33.909084 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:33.909671 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:33.909684 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:33.911675 (+ 1991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":98,"threads_started":1}
W20260925 22:41:33.911913 11501 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:33.906169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:43057 (local address 127.2.74.254:41871)
0925 22:41:33.906347 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:33.906352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:33.906508 (+ 156us) server_negotiation.cc:407] Connection header received
0925 22:41:33.906685 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:33.906688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:33.906741 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:33.906828 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:33.907309 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:33.907834 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:33.908647 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:33.908871 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:33.908982 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:33.911804 (+ 2822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.195:43057: BlockingRecv error: recv got EOF from 127.2.74.195:43057 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 22:41:34 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:34 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:34.501461 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:34.495620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.193:58747)
0925 22:41:34.495913 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:34.495932 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:34.496052 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:34.496485 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:34.496487 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:34.496683 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:34.496785 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:34.496790 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:34.497619 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:34.497622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:34.498212 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:34.498222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:34.498649 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:34.498652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:34.498714 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:34.499367 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:34.499379 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:34.501333 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":70,"threads_started":1}
W20260925 22:41:34.501672 11505 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:34.495947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:58747 (local address 127.2.74.254:41871)
0925 22:41:34.496211 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:34.496214 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:34.496227 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:41:34.496267 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:34.496269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:34.496312 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:34.496443 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:34.496926 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:34.497477 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:34.498325 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:34.498544 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:34.498624 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:34.501501 (+ 2877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.193:58747: BlockingRecv error: recv got EOF from 127.2.74.193:58747 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":107,"threads_started":1}
Sep 25 22:41:34 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:34 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:34.713125 11506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:34.707493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.194:54955)
0925 22:41:34.707756 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:34.707769 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:34.707903 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:34.708199 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:34.708201 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:34.708378 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:34.708478 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:34.708483 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:34.709222 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:34.709225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:34.709789 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:34.709798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:34.710271 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:34.710274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:34.710354 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:34.710894 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:34.710906 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:34.712948 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":76,"threads_started":1}
W20260925 22:41:34.713215 11505 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:34.707546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:54955 (local address 127.2.74.254:41871)
0925 22:41:34.707683 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:34.707689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:34.707857 (+ 168us) server_negotiation.cc:407] Connection header received
0925 22:41:34.708013 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:34.708017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:34.708071 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:34.708175 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:34.708578 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:34.709107 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:34.709915 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:34.710133 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:34.710219 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:34.713088 (+ 2869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.194:54955: BlockingRecv error: recv got EOF from 127.2.74.194:54955 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 22:41:34 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:34 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:34.918495 11507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:34.912736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.195:36815)
0925 22:41:34.913025 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:34.913038 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:34.913169 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:34.913462 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:34.913465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:34.913668 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:34.913762 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:34.913768 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:34.914530 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:34.914535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:34.915141 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:34.915151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:34.915630 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:34.915633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:34.915717 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:34.916253 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:34.916273 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:34.918312 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":85,"threads_started":1}
W20260925 22:41:34.918562 11505 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:34.912795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:36815 (local address 127.2.74.254:41871)
0925 22:41:34.912961 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:34.912966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:34.913109 (+ 143us) server_negotiation.cc:407] Connection header received
0925 22:41:34.913283 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:34.913288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:34.913341 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:34.913437 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:34.913889 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:34.914417 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:34.915271 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:34.915513 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:34.915624 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:34.918451 (+ 2827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.195:36815: BlockingRecv error: recv got EOF from 127.2.74.195:36815 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 22:41:35 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:35 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:35.508333 11508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:35.502509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.193:50583)
0925 22:41:35.502789 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:35.502803 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:35.502894 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:35.503197 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:35.503199 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:35.503377 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:35.503478 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:35.503483 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:35.504197 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:35.504199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:35.504877 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:35.504887 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:35.505502 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:35.505506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:35.505592 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:35.506152 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:35.506164 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:35.508217 (+ 2053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":92,"threads_started":1}
W20260925 22:41:35.508534 11509 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:35.502573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:50583 (local address 127.2.74.254:41871)
0925 22:41:35.502845 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:35.502849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:35.502865 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:41:35.503013 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:35.503016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:35.503072 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:35.503154 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:35.503583 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:35.504100 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:35.505112 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:35.505381 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:35.505497 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:35.508357 (+ 2860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.193:50583: BlockingRecv error: recv got EOF from 127.2.74.193:50583 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":82,"threads_started":1}
Sep 25 22:41:35 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:35 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:35.719789 11510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:35.714157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.194:48635)
0925 22:41:35.714451 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:35.714471 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:35.714565 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:35.714857 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:35.714859 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:35.715096 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:35.715194 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:35.715200 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:35.715957 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:35.715959 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:35.716508 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:35.716515 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:35.716909 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:35.716911 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:35.716971 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:35.717541 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:35.717554 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:35.719672 (+ 2118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":110,"threads_started":1}
W20260925 22:41:35.719976 11509 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:35.714297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:48635 (local address 127.2.74.254:41871)
0925 22:41:35.714456 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:35.714463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:35.714544 (+ 81us) server_negotiation.cc:407] Connection header received
0925 22:41:35.714659 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:35.714662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:35.714715 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:35.714819 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:35.715311 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:35.715858 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:35.716603 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:35.716810 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:35.716884 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:35.719818 (+ 2934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.194:48635: BlockingRecv error: recv got EOF from 127.2.74.194:48635 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 22:41:35 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:35 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:35.925235 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:35.919490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.195:39511)
0925 22:41:35.919858 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:35.919874 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:35.919992 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:35.920244 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:35.920246 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:35.920478 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:35.920630 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:35.920637 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:35.921439 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:35.921442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:35.922048 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:35.922056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:35.922483 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:35.922486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:35.922550 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:35.923286 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:35.923297 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:35.925118 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":102,"threads_started":1}
W20260925 22:41:35.925390 11509 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:35.919598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:39511 (local address 127.2.74.254:41871)
0925 22:41:35.919759 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:35.919764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:35.919940 (+ 176us) server_negotiation.cc:407] Connection header received
0925 22:41:35.920085 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:35.920088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:35.920135 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:35.920199 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:35.920745 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:35.921321 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:35.922162 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:35.922381 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:35.922459 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:35.925269 (+ 2810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.195:39511: BlockingRecv error: recv got EOF from 127.2.74.195:39511 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 22:41:36 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:36 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:36.514685 11512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:36.509200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.193:39909)
0925 22:41:36.509528 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:36.509543 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:36.509655 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:36.509873 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:36.509875 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:36.510051 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:36.510153 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:36.510158 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:36.510886 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:36.510889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:36.511653 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:36.511662 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:36.512082 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:36.512085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:36.512151 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:36.512708 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:36.512729 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:36.514556 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":103,"threads_started":1}
W20260925 22:41:36.514895 11513 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:36.509245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:39909 (local address 127.2.74.254:41871)
0925 22:41:36.509520 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:36.509524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:36.509605 (+ 81us) server_negotiation.cc:407] Connection header received
0925 22:41:36.509721 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:36.509724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:36.509766 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:36.509840 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:36.510250 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:36.510778 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:36.511766 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:36.511986 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:36.512066 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:36.514720 (+ 2654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.193:39909: BlockingRecv error: recv got EOF from 127.2.74.193:39909 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":102,"threads_started":1}
Sep 25 22:41:36 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:36 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:36.726310 11514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:36.720627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.194:49777)
0925 22:41:36.720902 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:36.720915 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:36.721015 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:36.721255 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:36.721258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:36.721436 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:36.721532 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:36.721537 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:36.722343 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:36.722346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:36.723229 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:36.723242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:36.723689 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:36.723695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:36.723787 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:36.724330 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:36.724342 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:36.726190 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":104,"threads_started":1}
W20260925 22:41:36.726594 11513 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:36.720692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:49777 (local address 127.2.74.254:41871)
0925 22:41:36.720845 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:36.720849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:36.720987 (+ 138us) server_negotiation.cc:407] Connection header received
0925 22:41:36.721098 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:36.721101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:36.721145 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:36.721224 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:36.721661 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:36.722172 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:36.723357 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:36.723578 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:36.723669 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:36.726402 (+ 2733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.194:49777: BlockingRecv error: recv got EOF from 127.2.74.194:49777 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 22:41:36 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:36 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:36.932300 11515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:36.926184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.195:45933)
0925 22:41:36.926601 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:36.926626 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:36.926762 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:36.927075 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:36.927078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:36.927305 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:36.927397 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:36.927402 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:36.928238 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:36.928243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:36.928846 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:36.928854 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:36.929348 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:36.929352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:36.929440 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:36.930012 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:36.930024 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:36.932103 (+ 2079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":121,"threads_started":1}
W20260925 22:41:36.932365 11513 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:36.926266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:45933 (local address 127.2.74.254:41871)
0925 22:41:36.926434 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:36.926439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:36.926710 (+ 271us) server_negotiation.cc:407] Connection header received
0925 22:41:36.926876 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:36.926880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:36.926938 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:36.927050 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:36.927526 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:36.928119 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:36.928974 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:36.929226 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:36.929341 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:36.932254 (+ 2913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.195:45933: BlockingRecv error: recv got EOF from 127.2.74.195:45933 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 22:41:37 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:37 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:37.521888 11516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:37.515558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.193:40271)
0925 22:41:37.515937 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:37.515950 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:37.516095 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:37.516378 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:37.516381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:37.516595 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:37.516685 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:37.516691 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:37.517591 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:37.517596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:37.518278 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:37.518296 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:37.518871 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:37.518874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:37.518989 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:37.519627 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:37.519676 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:37.521765 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":69,"threads_started":1}
W20260925 22:41:37.522076 11517 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:37.515617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:40271 (local address 127.2.74.254:41871)
0925 22:41:37.515879 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:37.515884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:37.516031 (+ 147us) server_negotiation.cc:407] Connection header received
0925 22:41:37.516199 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:37.516202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:37.516258 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:37.516359 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:37.516811 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:37.517481 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:37.518416 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:37.518761 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:37.518865 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:37.521927 (+ 3062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.193:40271: BlockingRecv error: recv got EOF from 127.2.74.193:40271 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":91,"threads_started":1}
Sep 25 22:41:37 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:37 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:37.733197 11518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:37.727355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.194:43381)
0925 22:41:37.727659 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:37.727678 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:37.727813 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:37.728127 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:37.728129 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:37.728329 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:37.728432 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:37.728438 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:37.729362 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:37.729364 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:37.730028 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:37.730037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:37.730476 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:37.730478 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:37.730539 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:37.731075 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:37.731088 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:37.733062 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":110,"threads_started":1}
W20260925 22:41:37.733520 11517 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:37.727445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:43381 (local address 127.2.74.254:41871)
0925 22:41:37.727593 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:37.727596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:37.727802 (+ 206us) server_negotiation.cc:407] Connection header received
0925 22:41:37.727889 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:37.727900 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:37.727949 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:37.728029 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:37.728673 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:37.729236 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:37.730144 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:37.730367 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:37.730449 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:37.733271 (+ 2822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.194:43381: BlockingRecv error: recv got EOF from 127.2.74.194:43381 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 22:41:37 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:37 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:37.942345 11519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:37.933336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.195:51311)
0925 22:41:37.933662 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:37.933677 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:37.933798 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:37.934081 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:37.934083 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:37.934281 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:37.934379 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:37.934385 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:37.935162 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:37.935166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:37.935737 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:37.935746 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:37.936197 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:37.936200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:37.936263 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:37.940124 (+ 3861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:37.940139 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:37.942163 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":70,"threads_started":1}
W20260925 22:41:37.942425 11517 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:37.933509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:51311 (local address 127.2.74.254:41871)
0925 22:41:37.933687 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:37.933691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:37.933758 (+ 67us) server_negotiation.cc:407] Connection header received
0925 22:41:37.933889 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:37.933891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:37.933939 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:37.934014 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:37.934509 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:37.935067 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:37.935862 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:37.936084 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:37.936171 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:37.942310 (+ 6139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.195:51311: BlockingRecv error: recv got EOF from 127.2.74.195:51311 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 25 22:41:38 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:38 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:38.532578 11520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:38.522757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.193:47547)
0925 22:41:38.523088 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:38.523102 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:38.523211 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:38.523522 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:38.523525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:38.523718 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:38.523820 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:38.523826 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:38.524602 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:38.524604 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:38.525174 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:38.525183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:38.525688 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:38.525691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:38.525766 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:38.530129 (+ 4363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:38.530144 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:38.532399 (+ 2255us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":72,"threads_started":1}
W20260925 22:41:38.532691 11521 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:38.522838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:47547 (local address 127.2.74.254:41871)
0925 22:41:38.523113 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:38.523116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:38.523273 (+ 157us) server_negotiation.cc:407] Connection header received
0925 22:41:38.523321 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:38.523323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:38.523369 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:38.523452 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:38.523945 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:38.524484 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:38.525298 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:38.525523 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:38.525608 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:38.532556 (+ 6948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.193:47547: BlockingRecv error: recv got EOF from 127.2.74.193:47547 (error 108)
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":126,"threads_started":1}
Sep 25 22:41:38 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:38 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:38.740899 11522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:38.734085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.194:37319)
0925 22:41:38.734394 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:38.734409 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:38.734517 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:38.734787 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:38.734789 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:38.735002 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:38.735103 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:38.735108 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:38.736124 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:38.736127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:38.736802 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:38.736813 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:38.737354 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:38.737357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:38.737432 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:38.738193 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:38.738206 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:38.740719 (+ 2513us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":131,"threads_started":1}
W20260925 22:41:38.741065 11521 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:38.734223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:37319 (local address 127.2.74.254:41871)
0925 22:41:38.734372 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:38.734376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:38.734483 (+ 107us) server_negotiation.cc:407] Connection header received
0925 22:41:38.734603 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:38.734605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:38.734655 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:38.734738 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:38.735305 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:38.735981 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:38.736967 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:38.737206 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:38.737302 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:38.740925 (+ 3623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.194:37319: BlockingRecv error: recv got EOF from 127.2.74.194:37319 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 22:41:38 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:38 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:38.948624 11523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:38.943304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.195:54443)
0925 22:41:38.943573 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:38.943589 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:38.943707 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:38.943981 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:38.943984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:38.944166 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:38.944272 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:38.944277 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:38.945002 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:38.945004 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:38.945571 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:38.945578 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:38.945993 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:38.945996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:38.946056 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:38.946549 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:38.946560 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:38.948481 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":100,"threads_started":1}
W20260925 22:41:38.948778 11521 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:38.943372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:54443 (local address 127.2.74.254:41871)
0925 22:41:38.943512 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:38.943517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:38.943650 (+ 133us) server_negotiation.cc:407] Connection header received
0925 22:41:38.943815 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:38.943820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:38.943881 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:38.943974 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:38.944378 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:38.944903 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:38.945672 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:38.945894 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:38.945971 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:38.948617 (+ 2646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.195:54443: BlockingRecv error: recv got EOF from 127.2.74.195:54443 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 22:41:39 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:39 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:39.538717 11524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:39.533452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.193:44995)
0925 22:41:39.533700 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:39.533722 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:39.533822 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:39.534080 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:39.534082 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:39.534272 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:39.534370 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:39.534376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:39.535110 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:39.535112 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:39.535677 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:39.535684 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:39.536093 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:39.536096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:39.536164 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:39.536751 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:39.536763 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:39.538598 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":91,"threads_started":1}
W20260925 22:41:39.538925 11525 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:39.533487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:44995 (local address 127.2.74.254:41871)
0925 22:41:39.533719 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:39.533723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:39.533772 (+ 49us) server_negotiation.cc:407] Connection header received
0925 22:41:39.533887 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:39.533889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:39.533931 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:39.534008 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:39.534475 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:39.535014 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:39.535776 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:39.535993 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:39.536070 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:39.538741 (+ 2671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.193:44995: BlockingRecv error: recv got EOF from 127.2.74.193:44995 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":85,"threads_started":1}
Sep 25 22:41:39 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:39 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:39.747926 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:39.741920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.194:38369)
0925 22:41:39.742228 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:39.742242 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:39.742344 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:39.742603 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:39.742605 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:39.742792 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:39.742888 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:39.742894 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:39.743770 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:39.743773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:39.744352 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:39.744360 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:39.744840 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:39.744843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:39.744911 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:39.745472 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:39.745484 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:39.747733 (+ 2249us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":120,"threads_started":1}
W20260925 22:41:39.748051 11525 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:39.742061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:38369 (local address 127.2.74.254:41871)
0925 22:41:39.742215 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:39.742218 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:39.742316 (+ 98us) server_negotiation.cc:407] Connection header received
0925 22:41:39.742431 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:39.742433 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:39.742478 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:39.742564 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:39.743074 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:39.743626 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:39.744479 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:39.744709 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:39.744796 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:39.747916 (+ 3120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.194:38369: BlockingRecv error: recv got EOF from 127.2.74.194:38369 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 22:41:39 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:39 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:39.956344 11527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:39.949517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.195:57607)
0925 22:41:39.949900 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:39.949916 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:39.950069 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:39.950373 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:39.950376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:39.950592 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:39.950726 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:39.950732 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:39.951699 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:39.951704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:39.952520 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:39.952537 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:39.953249 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:39.953255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:39.953384 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:39.953998 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:39.954010 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:39.956195 (+ 2185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":113,"threads_started":1}
W20260925 22:41:39.956496 11525 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:39.949700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:57607 (local address 127.2.74.254:41871)
0925 22:41:39.949869 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:39.949873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:39.950008 (+ 135us) server_negotiation.cc:407] Connection header received
0925 22:41:39.950203 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:39.950206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:39.950252 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:39.950346 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:39.950860 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:39.951579 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:39.952696 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:39.953096 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:39.953216 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:39.956369 (+ 3153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.195:57607: BlockingRecv error: recv got EOF from 127.2.74.195:57607 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 22:41:40 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:40 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:40.545696 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:40.539697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.193:38429)
0925 22:41:40.540048 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:40.540063 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:40.540159 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:40.540485 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:40.540491 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:40.540726 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:40.540884 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:40.540893 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:40.541935 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:40.541938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:40.542508 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:40.542516 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:40.542995 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:40.543002 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:40.543088 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:40.543663 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:40.543675 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:40.545556 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":113,"threads_started":1}
W20260925 22:41:40.545938 11529 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:40.539871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:38429 (local address 127.2.74.254:41871)
0925 22:41:40.540105 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:40.540110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:40.540130 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:41:40.540280 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:40.540284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:40.540340 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:40.540421 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:40.541021 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:40.541831 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:40.542609 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:40.542848 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:40.542924 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:40.545772 (+ 2848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.193:38429: BlockingRecv error: recv got EOF from 127.2.74.193:38429 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":60,"threads_started":1}
Sep 25 22:41:40 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:40 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:40.754004 11530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:40.748964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.194:48511)
0925 22:41:40.749269 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:40.749283 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:40.749377 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:40.749622 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:40.749624 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:40.749812 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:40.749908 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:40.749914 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:40.750617 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:40.750619 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:40.751209 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:40.751216 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:40.751615 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:40.751617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:40.751675 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:40.752163 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:40.752175 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:40.753885 (+ 1710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":125,"threads_started":1}
W20260925 22:41:40.754138 11529 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:40.749096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:48511 (local address 127.2.74.254:41871)
0925 22:41:40.749241 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:40.749245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:40.749351 (+ 106us) server_negotiation.cc:407] Connection header received
0925 22:41:40.749458 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:40.749467 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:40.749511 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:40.749585 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:40.750009 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:40.750523 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:40.751306 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:40.751517 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:40.751592 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:40.754023 (+ 2431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.194:48511: BlockingRecv error: recv got EOF from 127.2.74.194:48511 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 22:41:40 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:40 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:40.963668 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:40.957550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.195:35909)
0925 22:41:40.957898 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:40.957915 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:40.958008 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:40.958247 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:40.958249 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:40.958436 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:40.958585 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:40.958592 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:40.959672 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:40.959679 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:40.960627 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:40.960638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:40.961116 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:40.961118 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:40.961181 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:40.961702 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:40.961713 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:40.963546 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":93,"threads_started":1}
W20260925 22:41:40.963817 11529 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:40.957775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:35909 (local address 127.2.74.254:41871)
0925 22:41:40.957940 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:40.957944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:40.957982 (+ 38us) server_negotiation.cc:407] Connection header received
0925 22:41:40.958087 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:40.958089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:40.958136 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:40.958253 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:40.958747 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:40.959561 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:40.960753 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:40.961010 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:40.961092 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:40.963693 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.195:35909: BlockingRecv error: recv got EOF from 127.2.74.195:35909 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 25 22:41:41 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:41 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:41.552520 11532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:41.546670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.193:49219)
0925 22:41:41.547002 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:41.547019 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:41.547142 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:41.547499 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:41.547502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:41.547738 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:41.547882 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:41.547890 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:41.548769 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:41.548772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:41.549342 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:41.549351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:41.549776 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:41.549778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:41.549839 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:41.550451 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:41.550474 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:41.552392 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":89,"threads_started":1}
W20260925 22:41:41.552731 11533 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:41.546773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:49219 (local address 127.2.74.254:41871)
0925 22:41:41.547080 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:41.547083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:41.547114 (+ 31us) server_negotiation.cc:407] Connection header received
0925 22:41:41.547257 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:41.547261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:41.547322 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:41.547433 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:41.548018 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:41.548634 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:41.549454 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:41.549669 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:41.549746 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:41.552554 (+ 2808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.193:49219: BlockingRecv error: recv got EOF from 127.2.74.193:49219 (error 108)
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":71,"threads_started":1}
Sep 25 22:41:41 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:41 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:41.761102 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:41.755152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.194:47705)
0925 22:41:41.755619 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:41.755654 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:41.755817 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:41.756105 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:41.756108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:41.756318 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:41.756411 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:41.756417 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:41.757179 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:41.757186 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:41.757808 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:41.757817 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:41.758300 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:41.758303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:41.758391 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:41.758933 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:41.758944 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:41.760920 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":95,"threads_started":1}
W20260925 22:41:41.761193 11533 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:41.755319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:47705 (local address 127.2.74.254:41871)
0925 22:41:41.755594 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:41.755599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:41.755753 (+ 154us) server_negotiation.cc:407] Connection header received
0925 22:41:41.755925 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:41.755928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:41.755986 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:41.756081 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:41.756543 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:41.757069 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:41.757939 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:41.758168 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:41.758285 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:41.761071 (+ 2786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.194:47705: BlockingRecv error: recv got EOF from 127.2.74.194:47705 (error 108)
Metrics: {"server-negotiator.queue_time_us":166}
Sep 25 22:41:41 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:41 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:41.970939 11535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:41.964577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.195:37885)
0925 22:41:41.964958 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:41.964976 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:41.965140 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:41.965433 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:41.965437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:41.965661 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:41.965756 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:41.965762 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:41.966628 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:41.966634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:41.967501 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:41.967512 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:41.968014 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:41.968017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:41.968083 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:41.968620 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:41.968632 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:41.970767 (+ 2135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":110,"threads_started":1}
W20260925 22:41:41.971081 11533 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:41.964757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:37885 (local address 127.2.74.254:41871)
0925 22:41:41.964940 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:41.964945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:41.965077 (+ 132us) server_negotiation.cc:407] Connection header received
0925 22:41:41.965250 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:41.965254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:41.965312 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:41.965410 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:41.965884 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:41.966514 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:41.967640 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:41.967908 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:41.967999 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:41.970928 (+ 2929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.195:37885: BlockingRecv error: recv got EOF from 127.2.74.195:37885 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 25 22:41:42 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:42 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:42.559556 11536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:42.553526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.193:47637)
0925 22:41:42.553831 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:42.553846 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:42.553950 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:42.554221 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:42.554223 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:42.554415 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:42.554515 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:42.554520 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:42.555496 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:42.555500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:42.556199 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:42.556211 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:42.556777 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:42.556781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:42.556872 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:42.557373 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:42.557384 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:42.559372 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":80,"threads_started":1}
W20260925 22:41:42.559648 11537 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:42.553599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:47637 (local address 127.2.74.254:41871)
0925 22:41:42.553903 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:42.553907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:42.553923 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:41:42.554050 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:42.554052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:42.554098 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:42.554187 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:42.554623 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:42.555366 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:42.556331 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:42.556647 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:42.556760 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:42.559521 (+ 2761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.193:47637: BlockingRecv error: recv got EOF from 127.2.74.193:47637 (error 108)
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":110,"threads_started":1}
Sep 25 22:41:42 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:42 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:42.767582 11538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:42.761924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.194:50237)
0925 22:41:42.762232 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:42.762246 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:42.762357 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:42.762588 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:42.762590 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:42.762774 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:42.762871 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:42.762877 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:42.763743 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:42.763747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:42.764315 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:42.764324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:42.764820 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:42.764824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:42.764894 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:42.765456 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:42.765468 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:42.767459 (+ 1991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":79,"threads_started":1}
W20260925 22:41:42.767915 11537 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:42.762102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:50237 (local address 127.2.74.254:41871)
0925 22:41:42.762234 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:42.762238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:42.762303 (+ 65us) server_negotiation.cc:407] Connection header received
0925 22:41:42.762429 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:42.762431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:42.762474 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:42.762542 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:42.763072 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:42.763615 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:42.764450 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:42.764675 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:42.764778 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:42.767628 (+ 2850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.194:50237: BlockingRecv error: recv got EOF from 127.2.74.194:50237 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 22:41:42 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:42 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:42.977979 11539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:42.971923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.195:59715)
0925 22:41:42.972308 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:42.972324 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:42.972493 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:42.972794 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:42.972797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:42.973030 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:42.973126 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:42.973133 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:42.973923 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:42.973928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:42.974508 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:42.974516 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:42.975032 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:42.975036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:42.975124 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:42.975782 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:42.975795 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:42.977851 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":94,"threads_started":1}
W20260925 22:41:42.978202 11537 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:42.972064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:59715 (local address 127.2.74.254:41871)
0925 22:41:42.972272 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:42.972277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:42.972428 (+ 151us) server_negotiation.cc:407] Connection header received
0925 22:41:42.972605 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:42.972608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:42.972667 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:42.972768 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:42.973259 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:42.973796 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:42.974641 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:42.974872 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:42.975014 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:42.978053 (+ 3039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.195:59715: BlockingRecv error: recv got EOF from 127.2.74.195:59715 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 25 22:41:43 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:43 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:43.572793 11540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:43.560467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.193:44277)
0925 22:41:43.560756 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:43.560771 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:43.560869 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:43.561156 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:43.561158 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:43.561345 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:43.561450 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:43.561455 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:43.562264 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:43.562267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:43.562840 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:43.562849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:43.563392 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:43.563395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:43.563461 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:43.570311 (+ 6850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:43.570334 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:43.572556 (+ 2222us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":99,"threads_started":1}
W20260925 22:41:43.572841 11541 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:43.560550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:44277 (local address 127.2.74.254:41871)
0925 22:41:43.560822 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:43.560826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:43.560842 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:41:43.560980 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:43.560983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:43.561030 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:43.561117 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:43.561568 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:43.562132 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:43.563022 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:43.563274 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:43.563365 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:43.572716 (+ 9351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.193:44277: BlockingRecv error: recv got EOF from 127.2.74.193:44277 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":63,"threads_started":1}
Sep 25 22:41:43 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:43 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:43.774616 11542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:43.768567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.194:43039)
0925 22:41:43.768903 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:43.768918 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:43.769030 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:43.769280 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:43.769282 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:43.769529 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:43.769631 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:43.769637 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:43.770516 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:43.770518 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:43.771294 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:43.771304 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:43.771824 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:43.771827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:43.771926 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:43.772536 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:43.772549 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:43.774422 (+ 1873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":96,"threads_started":1}
W20260925 22:41:43.774674 11541 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:43.768772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:43039 (local address 127.2.74.254:41871)
0925 22:41:43.768943 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:43.768947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:43.768979 (+ 32us) server_negotiation.cc:407] Connection header received
0925 22:41:43.769116 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:43.769119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:43.769165 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:43.769242 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:43.769747 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:43.770411 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:43.771426 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:43.771690 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:43.771796 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:43.774560 (+ 2764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.194:43039: BlockingRecv error: recv got EOF from 127.2.74.194:43039 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 25 22:41:43 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:43 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:43.984819 11543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:43.978816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.195:40659)
0925 22:41:43.979155 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:43.979177 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:43.979324 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:43.979623 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:43.979626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:43.979857 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:43.979952 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:43.979957 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:43.980742 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:43.980748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:43.981343 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:43.981352 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:43.981855 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:43.981859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:43.981946 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:43.982552 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:43.982564 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:43.984624 (+ 2060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":108,"threads_started":1}
W20260925 22:41:43.984936 11541 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:43.978981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:40659 (local address 127.2.74.254:41871)
0925 22:41:43.979123 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:43.979127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:43.979270 (+ 143us) server_negotiation.cc:407] Connection header received
0925 22:41:43.979448 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:43.979451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:43.979508 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:43.979597 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:43.980081 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:43.980615 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:43.981473 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:43.981731 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:43.981849 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:43.984821 (+ 2972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.195:40659: BlockingRecv error: recv got EOF from 127.2.74.195:40659 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 22:41:44 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:44 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:44.579782 11544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:44.573903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.193:36953)
0925 22:41:44.574210 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:44.574227 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:44.574352 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:44.574613 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:44.574615 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:44.574801 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:44.574903 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:44.574908 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:44.575723 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:44.575726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:44.576315 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:44.576324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:44.576781 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:44.576783 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:44.576847 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:44.577479 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:44.577492 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:44.579658 (+ 2166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":75,"threads_started":1}
W20260925 22:41:44.579988 11545 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:44.573970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:36953 (local address 127.2.74.254:41871)
0925 22:41:44.574326 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:44.574329 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:44.574344 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:41:44.574445 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:44.574448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:44.574492 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:44.574573 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:44.575074 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:44.575608 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:44.576444 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:44.576672 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:44.576756 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:44.579811 (+ 3055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.193:36953: BlockingRecv error: recv got EOF from 127.2.74.193:36953 (error 108)
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":95,"threads_started":1}
Sep 25 22:41:44 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:44 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:44.781373 11546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:44.775567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.194:34943)
0925 22:41:44.775876 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:44.775899 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:44.776004 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:44.776278 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:44.776280 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:44.776472 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:44.776581 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:44.776587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:44.777355 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:44.777357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:44.777937 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:44.777944 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:44.778398 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:44.778400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:44.778480 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:44.779110 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:44.779125 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:44.781161 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":132,"threads_started":1}
W20260925 22:41:44.781425 11545 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:44.775702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:34943 (local address 127.2.74.254:41871)
0925 22:41:44.775875 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:44.775879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:44.775975 (+ 96us) server_negotiation.cc:407] Connection header received
0925 22:41:44.776097 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:44.776099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:44.776144 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:44.776230 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:44.776705 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:44.777243 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:44.778053 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:44.778285 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:44.778366 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:44.781312 (+ 2946us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.194:34943: BlockingRecv error: recv got EOF from 127.2.74.194:34943 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 22:41:44 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:44 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.195@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:44.992385 11547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:44.985945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.195:60869)
0925 22:41:44.986255 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:44.986270 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:44.986376 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:44.986794 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:44.986797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:44.987064 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:44.987167 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:44.987175 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:44.988163 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:44.988166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:44.988855 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:44.988866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:44.989367 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:44.989377 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:44.989449 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:44.990117 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:44.990130 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:44.992243 (+ 2113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":95,"threads_started":1}
W20260925 22:41:44.992573 11545 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:44.986093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:60869 (local address 127.2.74.254:41871)
0925 22:41:44.986298 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:44.986302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:44.986321 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:41:44.986547 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:44.986551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:44.986613 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:44.986714 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:44.987283 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:44.988035 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:44.988965 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:44.989185 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:44.989275 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:44.992413 (+ 3138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.195:60869: BlockingRecv error: recv got EOF from 127.2.74.195:60869 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 25 22:41:45 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:45 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.193@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:45.586607 11548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:45.580668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.193:56079)
0925 22:41:45.580945 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:45.580963 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:45.581068 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:45.581312 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:45.581314 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:45.581497 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:45.581595 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:45.581601 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:45.582405 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:45.582408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:45.583136 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:45.583149 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:45.583604 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:45.583607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:45.583671 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:45.584308 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:45.584321 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:45.586467 (+ 2146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":71,"threads_started":1}
W20260925 22:41:45.586874 11549 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:45.580805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:56079 (local address 127.2.74.254:41871)
0925 22:41:45.581064 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:45.581068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:45.581081 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:41:45.581143 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:45.581145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:45.581186 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:45.581269 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:45.581726 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:45.582274 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:45.583261 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:45.583488 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:45.583573 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:45.586679 (+ 3106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.193:56079: BlockingRecv error: recv got EOF from 127.2.74.193:56079 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":96,"threads_started":1}
Sep 25 22:41:45 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:41:45 dist-test-slave-csnl krb5kdc[10490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.74.194@KRBTEST.COM for oryx/127.2.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:41:45.788579 11550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:45.782606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:41871 (local address 127.2.74.194:60467)
0925 22:41:45.783001 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:45.783030 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:45.783146 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:45.783479 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:45.783483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:45.783716 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:45.783844 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:45.783851 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:45.784712 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:45.784717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:45.785363 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:45.785372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:45.785848 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:45.785851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:45.785918 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:45.786417 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:45.786428 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:45.788456 (+ 2028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.74.254:41871: Server oryx/127.2.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":91,"threads_started":1}
W20260925 22:41:45.788780 11549 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:45.782786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:60467 (local address 127.2.74.254:41871)
0925 22:41:45.782993 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:45.782997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:45.783148 (+ 151us) server_negotiation.cc:407] Connection header received
0925 22:41:45.783234 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:45.783237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:45.783307 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:45.783393 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:45.783942 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:45.784592 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:45.785487 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:45.785699 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:45.785786 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:45.788647 (+ 2861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.74.194:60467: BlockingRecv error: recv got EOF from 127.2.74.194:60467 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260925 22:41:45.868214 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 11073
I20260925 22:41:45.874591 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 11226
I20260925 22:41:45.880475 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 11358
I20260925 22:41:45.886445 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 11003
2026-09-25T22:41:45Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18116 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T22:41:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:41:45Z Disabled control of system clock
I20260925 22:41:45.918493 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:36179
--webserver_interface=127.2.74.254
--webserver_port=0
--builtin_ntp_servers=127.2.74.212:44555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:36179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 22:41:46.036855 11554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:46.037169 11554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:46.037206 11554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:46.041493 11554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:41:46.041579 11554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:46.041595 11554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:46.041612 11554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:41:46.041628 11554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:41:46.046904 11554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:44555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:36179
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:36179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:46.048153 11554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:46.049664 11554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:46.059316 11562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:46.059280 11554 server_base.cc:1034] running on GCE node
W20260925 22:41:46.059285 11559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:46.059273 11560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:46.060211 11554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:46.061452 11554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:46.062655 11554 hybrid_clock.cc:648] HybridClock initialized: now 1790376106062631 us; error 56 us; skew 500 ppm
I20260925 22:41:46.064934 11554 webserver.cc:460] Webserver started at http://127.2.74.254:35343/ using document root <none> and password file <none>
I20260925 22:41:46.065527 11554 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:46.065591 11554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:46.065769 11554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:46.068060 11554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "551290bd2e8d41359e4be019599d7b44"
format_stamp: "Formatted at 2026-09-25 22:41:46 on dist-test-slave-csnl"
server_key: "a9e9085063343601677c0115d4e586db"
server_key_iv: "4810ab544c83295ca4e1d11d05274dda"
server_key_version: "encryptionkey@0"
I20260925 22:41:46.068639 11554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "551290bd2e8d41359e4be019599d7b44"
format_stamp: "Formatted at 2026-09-25 22:41:46 on dist-test-slave-csnl"
server_key: "a9e9085063343601677c0115d4e586db"
server_key_iv: "4810ab544c83295ca4e1d11d05274dda"
server_key_version: "encryptionkey@0"
I20260925 22:41:46.072763 11554 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:41:46.075732 11568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:46.077111 11554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:41:46.077281 11554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "551290bd2e8d41359e4be019599d7b44"
format_stamp: "Formatted at 2026-09-25 22:41:46 on dist-test-slave-csnl"
server_key: "a9e9085063343601677c0115d4e586db"
server_key_iv: "4810ab544c83295ca4e1d11d05274dda"
server_key_version: "encryptionkey@0"
I20260925 22:41:46.077420 11554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:46.091979 11554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:46.092779 11554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:46.093005 11554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:46.100507 11554 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:36179
I20260925 22:41:46.100565 11620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:36179 every 8 connection(s)
I20260925 22:41:46.101619 11554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:41:46.105037 11621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:46.105803 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 11554
I20260925 22:41:46.105903 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:41:46.106243 2347 external_mini_cluster.cc:1428] Setting key 83c3227a491e1c2b4d562b3ffecfacf1
I20260925 22:41:46.112010 11621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 551290bd2e8d41359e4be019599d7b44: Bootstrap starting.
I20260925 22:41:46.115085 11621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 551290bd2e8d41359e4be019599d7b44: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:46.116048 11621 log.cc:824] T 00000000000000000000000000000000 P 551290bd2e8d41359e4be019599d7b44: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:46.117131 11624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:46.107883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35652 (local address 127.2.74.254:36179)
0925 22:41:46.108598 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:46.108615 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:46.108654 (+ 39us) server_negotiation.cc:407] Connection header received
0925 22:41:46.109354 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:46.109375 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:46.109844 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:46.110181 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:46.110584 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:46.111707 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:46.113036 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:46.113612 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:46.113773 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:46.114802 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:46.114816 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:46.114831 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:41:46.115430 (+ 599us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:46.115495 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:41:46.115518 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:46.115606 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:46.115791 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 22:41:46.116159 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":167,"threads_started":1}
I20260925 22:41:46.118387 11621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 551290bd2e8d41359e4be019599d7b44: No bootstrap required, opened a new log
I20260925 22:41:46.122663 11621 raft_consensus.cc:348] T 00000000000000000000000000000000 P 551290bd2e8d41359e4be019599d7b44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "551290bd2e8d41359e4be019599d7b44" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36179 } }
I20260925 22:41:46.122918 11621 raft_consensus.cc:374] T 00000000000000000000000000000000 P 551290bd2e8d41359e4be019599d7b44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:46.122999 11621 raft_consensus.cc:729] T 00000000000000000000000000000000 P 551290bd2e8d41359e4be019599d7b44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 551290bd2e8d41359e4be019599d7b44, State: Initialized, Role: FOLLOWER
I20260925 22:41:46.123414 11621 consensus_queue.cc:260] T 00000000000000000000000000000000 P 551290bd2e8d41359e4be019599d7b44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "551290bd2e8d41359e4be019599d7b44" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36179 } }
I20260925 22:41:46.123531 11621 raft_consensus.cc:388] T 00000000000000000000000000000000 P 551290bd2e8d41359e4be019599d7b44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:41:46.123572 11621 raft_consensus.cc:482] T 00000000000000000000000000000000 P 551290bd2e8d41359e4be019599d7b44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:41:46.123638 11621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 551290bd2e8d41359e4be019599d7b44 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:46.124681 11621 raft_consensus.cc:504] T 00000000000000000000000000000000 P 551290bd2e8d41359e4be019599d7b44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "551290bd2e8d41359e4be019599d7b44" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36179 } }
I20260925 22:41:46.124990 11621 leader_election.cc:302] T 00000000000000000000000000000000 P 551290bd2e8d41359e4be019599d7b44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 551290bd2e8d41359e4be019599d7b44; no voters:
I20260925 22:41:46.125263 11621 leader_election.cc:288] T 00000000000000000000000000000000 P 551290bd2e8d41359e4be019599d7b44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:41:46.125415 11626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 551290bd2e8d41359e4be019599d7b44 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:41:46.125667 11626 raft_consensus.cc:686] T 00000000000000000000000000000000 P 551290bd2e8d41359e4be019599d7b44 [term 1 LEADER]: Becoming Leader. State: Replica: 551290bd2e8d41359e4be019599d7b44, State: Running, Role: LEADER
I20260925 22:41:46.126022 11626 consensus_queue.cc:237] T 00000000000000000000000000000000 P 551290bd2e8d41359e4be019599d7b44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "551290bd2e8d41359e4be019599d7b44" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36179 } }
I20260925 22:41:46.126554 11621 sys_catalog.cc:564] T 00000000000000000000000000000000 P 551290bd2e8d41359e4be019599d7b44 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:41:46.127812 11627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 551290bd2e8d41359e4be019599d7b44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "551290bd2e8d41359e4be019599d7b44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "551290bd2e8d41359e4be019599d7b44" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36179 } } }
I20260925 22:41:46.127833 11628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 551290bd2e8d41359e4be019599d7b44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 551290bd2e8d41359e4be019599d7b44. Latest consensus state: current_term: 1 leader_uuid: "551290bd2e8d41359e4be019599d7b44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "551290bd2e8d41359e4be019599d7b44" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36179 } } }
I20260925 22:41:46.128033 11628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 551290bd2e8d41359e4be019599d7b44 [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:46.128433 11635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:41:46.129382 11627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 551290bd2e8d41359e4be019599d7b44 [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:46.131784 11635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:41:46.137902 11635 catalog_manager.cc:1347] Generated new cluster ID: 1804ee4ed92942d99971fa8d39990d1c
I20260925 22:41:46.138005 11635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:41:46.152956 11635 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:41:46.154342 11635 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:41:46.163270 11635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 551290bd2e8d41359e4be019599d7b44: Generated new TSK 0
I20260925 22:41:46.164212 11635 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:41:46.172106 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:0
--local_ip_for_outbound_sockets=127.2.74.193
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:36179
--builtin_ntp_servers=127.2.74.212:44555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 22:41:46.290711 11645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:46.291122 11645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:46.291206 11645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:46.295493 11645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:46.295615 11645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:46.295720 11645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:41:46.300791 11645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:44555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:36179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:46.302073 11645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:46.303656 11645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:46.314080 11650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:46.314206 11645 server_base.cc:1034] running on GCE node
W20260925 22:41:46.314061 11651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:46.314090 11653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:46.314800 11645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:46.315573 11645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:46.316813 11645 hybrid_clock.cc:648] HybridClock initialized: now 1790376106316773 us; error 59 us; skew 500 ppm
I20260925 22:41:46.318924 11645 webserver.cc:460] Webserver started at http://127.2.74.193:40561/ using document root <none> and password file <none>
I20260925 22:41:46.319621 11645 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:46.319723 11645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:46.319965 11645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:46.322168 11645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/data/instance:
uuid: "017dab53a04a46ceb7890c9a6b251941"
format_stamp: "Formatted at 2026-09-25 22:41:46 on dist-test-slave-csnl"
server_key: "a13b21fb162205bb058af5ca68ddea14"
server_key_iv: "52730c260df19acb7d91fe2509cfdccb"
server_key_version: "encryptionkey@0"
I20260925 22:41:46.322785 11645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance:
uuid: "017dab53a04a46ceb7890c9a6b251941"
format_stamp: "Formatted at 2026-09-25 22:41:46 on dist-test-slave-csnl"
server_key: "a13b21fb162205bb058af5ca68ddea14"
server_key_iv: "52730c260df19acb7d91fe2509cfdccb"
server_key_version: "encryptionkey@0"
I20260925 22:41:46.327035 11645 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 22:41:46.329949 11659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:46.331405 11645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 22:41:46.331568 11645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "017dab53a04a46ceb7890c9a6b251941"
format_stamp: "Formatted at 2026-09-25 22:41:46 on dist-test-slave-csnl"
server_key: "a13b21fb162205bb058af5ca68ddea14"
server_key_iv: "52730c260df19acb7d91fe2509cfdccb"
server_key_version: "encryptionkey@0"
I20260925 22:41:46.331736 11645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:46.354167 11645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:46.354902 11645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:46.355154 11645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:46.355748 11645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:46.357030 11645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:46.357085 11645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:46.357129 11645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:46.357146 11645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:46.366659 11645 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:44537
I20260925 22:41:46.366684 11772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:44537 every 8 connection(s)
I20260925 22:41:46.367759 11645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:41:46.369911 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 11645
I20260925 22:41:46.370054 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance
I20260925 22:41:46.370350 2347 external_mini_cluster.cc:1428] Setting key 8b110bd13c082f912fa0dfe042f7c03e
I20260925 22:41:46.372488 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:0
--local_ip_for_outbound_sockets=127.2.74.194
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:36179
--builtin_ntp_servers=127.2.74.212:44555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 22:41:46.379140 11624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:46.369944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:46115 (local address 127.2.74.254:36179)
0925 22:41:46.370864 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:46.370869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:46.371562 (+ 693us) server_negotiation.cc:407] Connection header received
0925 22:41:46.373216 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:46.373223 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:46.373293 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:46.373400 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:46.374806 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:46.375520 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:46.376827 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:46.377081 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:46.377208 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:46.378551 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:46.378555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:46.378557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:41:46.378572 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:46.378592 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:41:46.378600 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:46.378688 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:46.378872 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 22:41:46.378980 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807}
I20260925 22:41:46.380118 11775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:46.370503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:36179 (local address 127.2.74.193:46115)
0925 22:41:46.371373 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:46.371437 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:46.372957 (+ 1520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:46.373579 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:46.373587 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:46.373785 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:46.374599 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:46.374622 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:46.375652 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:46.375655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:46.376659 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:46.376670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:46.377261 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:46.377265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:46.377462 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:46.378206 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:41:46.378208 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:46.378259 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:41:46.378340 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:46.378690 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:46.378702 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:46.378711 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 22:41:46.378870 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 22:41:46.379240 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":163,"threads_started":1}
I20260925 22:41:46.381575 11773 heartbeater.cc:347] Connected to a master server at 127.2.74.254:36179
I20260925 22:41:46.381891 11773 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:46.382598 11773 heartbeater.cc:511] Master 127.2.74.254:36179 requested a full tablet report, sending...
I20260925 22:41:46.384367 11585 ts_manager.cc:194] Registered new tserver with Master: 017dab53a04a46ceb7890c9a6b251941 (127.2.74.193:44537)
I20260925 22:41:46.385596 11585 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.74.193:46115
W20260925 22:41:46.491047 11776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:46.491343 11776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:46.491382 11776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:46.495662 11776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:46.495739 11776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:46.495843 11776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:41:46.501056 11776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:44555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:36179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:46.502326 11776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:46.503857 11776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:46.514667 11781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:46.514712 11776 server_base.cc:1034] running on GCE node
W20260925 22:41:46.514648 11784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:46.514648 11782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:46.515405 11776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:46.516130 11776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:46.517355 11776 hybrid_clock.cc:648] HybridClock initialized: now 1790376106517313 us; error 53 us; skew 500 ppm
I20260925 22:41:46.519577 11776 webserver.cc:460] Webserver started at http://127.2.74.194:45799/ using document root <none> and password file <none>
I20260925 22:41:46.520298 11776 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:46.520411 11776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:46.520671 11776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:46.523126 11776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/data/instance:
uuid: "c78f9954506844c288c3b98926ce9d22"
format_stamp: "Formatted at 2026-09-25 22:41:46 on dist-test-slave-csnl"
server_key: "97a1c24048a6a9046fb5cd39d987ec5e"
server_key_iv: "f70e72f18eeebe52b6fcfd3ec700f23e"
server_key_version: "encryptionkey@0"
I20260925 22:41:46.523782 11776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance:
uuid: "c78f9954506844c288c3b98926ce9d22"
format_stamp: "Formatted at 2026-09-25 22:41:46 on dist-test-slave-csnl"
server_key: "97a1c24048a6a9046fb5cd39d987ec5e"
server_key_iv: "f70e72f18eeebe52b6fcfd3ec700f23e"
server_key_version: "encryptionkey@0"
I20260925 22:41:46.528373 11776 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 22:41:46.531298 11790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:46.532675 11776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260925 22:41:46.532828 11776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "c78f9954506844c288c3b98926ce9d22"
format_stamp: "Formatted at 2026-09-25 22:41:46 on dist-test-slave-csnl"
server_key: "97a1c24048a6a9046fb5cd39d987ec5e"
server_key_iv: "f70e72f18eeebe52b6fcfd3ec700f23e"
server_key_version: "encryptionkey@0"
I20260925 22:41:46.532961 11776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:46.557178 11776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:46.557963 11776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:46.558187 11776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:46.558853 11776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:46.559945 11776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:46.560020 11776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:46.560096 11776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:46.560140 11776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:46.569934 11776 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:37283
I20260925 22:41:46.569957 11903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:37283 every 8 connection(s)
I20260925 22:41:46.571067 11776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:41:46.580881 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 11776
I20260925 22:41:46.581012 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance
I20260925 22:41:46.581357 2347 external_mini_cluster.cc:1428] Setting key bd8be86a628c832e459fe713f3adc674
I20260925 22:41:46.581954 11624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:46.573274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:41565 (local address 127.2.74.254:36179)
0925 22:41:46.573423 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:46.573427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:46.574715 (+ 1288us) server_negotiation.cc:407] Connection header received
0925 22:41:46.576292 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:46.576295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:46.576353 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:46.576437 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:46.577530 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:46.578087 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:46.579511 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:46.579743 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:46.579851 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:46.581250 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:46.581253 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:46.581255 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:41:46.581270 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:46.581291 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:41:46.581299 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:46.581409 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:46.581659 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 22:41:46.581752 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:41:46.583010 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:46.573725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:36179 (local address 127.2.74.194:41565)
0925 22:41:46.574530 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:46.574575 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:46.576004 (+ 1429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:46.576648 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:46.576656 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:46.576769 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:46.577338 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:46.577354 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:46.578251 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:46.578254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:46.579377 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:46.579390 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:46.579917 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:46.579920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:46.580084 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:46.580875 (+ 791us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:41:46.580877 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:46.580944 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:41:46.581056 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:46.581464 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:46.581476 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:46.581483 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 22:41:46.581638 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 22:41:46.582047 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":162,"threads_started":1}
I20260925 22:41:46.583634 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:0
--local_ip_for_outbound_sockets=127.2.74.195
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:36179
--builtin_ntp_servers=127.2.74.212:44555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 22:41:46.584281 11904 heartbeater.cc:347] Connected to a master server at 127.2.74.254:36179
I20260925 22:41:46.584561 11904 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:46.585222 11904 heartbeater.cc:511] Master 127.2.74.254:36179 requested a full tablet report, sending...
I20260925 22:41:46.586483 11585 ts_manager.cc:194] Registered new tserver with Master: c78f9954506844c288c3b98926ce9d22 (127.2.74.194:37283)
I20260925 22:41:46.587404 11585 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.74.194:41565
W20260925 22:41:46.704296 11907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:46.704640 11907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:46.704681 11907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:46.709003 11907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:46.709091 11907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:46.709151 11907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:41:46.714335 11907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:44555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:36179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:46.715617 11907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:46.717109 11907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:41:46.727594 11907 server_base.cc:1034] running on GCE node
W20260925 22:41:46.727550 11915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:46.727555 11912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:46.727550 11913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:46.728268 11907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:46.728963 11907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:46.730127 11907 hybrid_clock.cc:648] HybridClock initialized: now 1790376106730105 us; error 43 us; skew 500 ppm
I20260925 22:41:46.732172 11907 webserver.cc:460] Webserver started at http://127.2.74.195:38035/ using document root <none> and password file <none>
I20260925 22:41:46.732806 11907 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:46.732877 11907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:46.733103 11907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:46.735577 11907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/data/instance:
uuid: "5a8b89355bee40b2bb364cec715cfdfa"
format_stamp: "Formatted at 2026-09-25 22:41:46 on dist-test-slave-csnl"
server_key: "238e1c29318e9bcc929fcfd9162aed9c"
server_key_iv: "56aca6bd8e7be9e8f846179283a4b2f7"
server_key_version: "encryptionkey@0"
I20260925 22:41:46.736271 11907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance:
uuid: "5a8b89355bee40b2bb364cec715cfdfa"
format_stamp: "Formatted at 2026-09-25 22:41:46 on dist-test-slave-csnl"
server_key: "238e1c29318e9bcc929fcfd9162aed9c"
server_key_iv: "56aca6bd8e7be9e8f846179283a4b2f7"
server_key_version: "encryptionkey@0"
I20260925 22:41:46.740803 11907 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 22:41:46.744019 11921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:46.745623 11907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:41:46.745826 11907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "5a8b89355bee40b2bb364cec715cfdfa"
format_stamp: "Formatted at 2026-09-25 22:41:46 on dist-test-slave-csnl"
server_key: "238e1c29318e9bcc929fcfd9162aed9c"
server_key_iv: "56aca6bd8e7be9e8f846179283a4b2f7"
server_key_version: "encryptionkey@0"
I20260925 22:41:46.745985 11907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:46.772168 11907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:46.772962 11907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:46.773187 11907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:46.773818 11907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:46.774820 11907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:46.774899 11907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:46.775003 11907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:46.775050 11907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:46.784634 11907 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:46709
I20260925 22:41:46.784653 12034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:46709 every 8 connection(s)
I20260925 22:41:46.785683 11907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:41:46.791503 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 11907
I20260925 22:41:46.791635 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance
I20260925 22:41:46.791935 2347 external_mini_cluster.cc:1428] Setting key 09a436031ba4b1e6b8b5e5f33c00c7b6
I20260925 22:41:46.795670 11624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:46.787781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:48081 (local address 127.2.74.254:36179)
0925 22:41:46.787952 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:46.787957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:46.789080 (+ 1123us) server_negotiation.cc:407] Connection header received
0925 22:41:46.790458 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:46.790461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:46.790508 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:46.790585 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:46.791640 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:46.792411 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:46.793555 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:46.793804 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:46.793944 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:46.795203 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:46.795206 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:46.795208 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:41:46.795220 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:46.795237 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:41:46.795245 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:46.795306 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:46.795500 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 22:41:46.795553 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 22:41:46.796645 12037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:46.788237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:36179 (local address 127.2.74.195:48081)
0925 22:41:46.788923 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:46.788973 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:46.790262 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:46.790754 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:46.790760 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:46.790864 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:46.791471 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:46.791485 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:46.792531 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:46.792534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:46.793425 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:46.793435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:46.793975 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:46.793978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:46.794199 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:46.794881 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:41:46.794882 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:46.794925 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:41:46.795049 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:46.795354 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:46.795363 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:46.795368 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:41:46.795490 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 22:41:46.795751 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":108,"threads_started":1}
I20260925 22:41:46.797982 12035 heartbeater.cc:347] Connected to a master server at 127.2.74.254:36179
I20260925 22:41:46.798207 12035 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:46.798717 12035 heartbeater.cc:511] Master 127.2.74.254:36179 requested a full tablet report, sending...
I20260925 22:41:46.799872 11585 ts_manager.cc:194] Registered new tserver with Master: 5a8b89355bee40b2bb364cec715cfdfa (127.2.74.195:46709)
I20260925 22:41:46.800410 11585 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.74.195:48081
I20260925 22:41:46.806025 2347 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 22:41:46.808871 11624 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:46.807965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35664 (local address 127.2.74.254:36179)
0925 22:41:46.808100 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:46.808104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:46.808294 (+ 190us) server_negotiation.cc:407] Connection header received
0925 22:41:46.808377 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:46.808379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:46.808422 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:46.808491 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:46.808698 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35664: BlockingRecv error: recv got EOF from 127.0.0.1:35664 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:41:46.809070 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 11645
I20260925 22:41:46.814450 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 11776
I20260925 22:41:46.819572 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 11907
I20260925 22:41:46.825094 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 11554
2026-09-25T22:41:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (936 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T22:41:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:41:46Z Disabled control of system clock
I20260925 22:41:46.855706 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:33325
--webserver_interface=127.2.74.254
--webserver_port=0
--builtin_ntp_servers=127.2.74.212:34375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:33325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 22:41:46.970929 12046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:46.971257 12046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:46.971297 12046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:46.975662 12046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:41:46.975744 12046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:46.975760 12046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:46.975778 12046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:41:46.975795 12046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:41:46.981633 12046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:34375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:33325
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:33325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:46.982895 12046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:46.984441 12046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:46.993319 12052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:46.993402 12051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:46.993430 12046 server_base.cc:1034] running on GCE node
W20260925 22:41:46.993319 12054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:46.994170 12046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:46.995427 12046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:46.996598 12046 hybrid_clock.cc:648] HybridClock initialized: now 1790376106996584 us; error 31 us; skew 500 ppm
I20260925 22:41:46.998606 12046 webserver.cc:460] Webserver started at http://127.2.74.254:44341/ using document root <none> and password file <none>
I20260925 22:41:46.999351 12046 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:46.999425 12046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:46.999655 12046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:47.001853 12046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "505f452f86ed42c09303eb68bc7f54ce"
format_stamp: "Formatted at 2026-09-25 22:41:46 on dist-test-slave-csnl"
server_key: "23b43bcaea20f006d25d0a2470f80399"
server_key_iv: "faec25dfabf5765a4738a0972479e33f"
server_key_version: "encryptionkey@0"
I20260925 22:41:47.002477 12046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "505f452f86ed42c09303eb68bc7f54ce"
format_stamp: "Formatted at 2026-09-25 22:41:46 on dist-test-slave-csnl"
server_key: "23b43bcaea20f006d25d0a2470f80399"
server_key_iv: "faec25dfabf5765a4738a0972479e33f"
server_key_version: "encryptionkey@0"
I20260925 22:41:47.006353 12046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:41:47.008999 12060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:47.010298 12046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:41:47.010449 12046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "505f452f86ed42c09303eb68bc7f54ce"
format_stamp: "Formatted at 2026-09-25 22:41:46 on dist-test-slave-csnl"
server_key: "23b43bcaea20f006d25d0a2470f80399"
server_key_iv: "faec25dfabf5765a4738a0972479e33f"
server_key_version: "encryptionkey@0"
I20260925 22:41:47.010571 12046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:47.015501 12046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:47.016197 12046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:47.016410 12046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:47.024106 12046 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:33325
I20260925 22:41:47.024114 12112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:33325 every 8 connection(s)
I20260925 22:41:47.025087 12046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:41:47.028586 12113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:47.032099 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 12046
I20260925 22:41:47.032219 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:41:47.032534 2347 external_mini_cluster.cc:1428] Setting key 099e11e0c00ada2cf877200e5ad229b3
I20260925 22:41:47.035297 12113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 505f452f86ed42c09303eb68bc7f54ce: Bootstrap starting.
I20260925 22:41:47.037798 12113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 505f452f86ed42c09303eb68bc7f54ce: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:47.038596 12113 log.cc:824] T 00000000000000000000000000000000 P 505f452f86ed42c09303eb68bc7f54ce: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:47.038619 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:47.034088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59976 (local address 127.2.74.254:33325)
0925 22:41:47.034797 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:47.034813 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:47.034876 (+ 63us) server_negotiation.cc:407] Connection header received
0925 22:41:47.035659 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:47.035680 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:47.036069 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:47.036439 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:47.037263 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:47.037273 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:47.037286 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:41:47.037296 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:47.037352 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:41:47.037369 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:47.037487 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:47.037589 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 22:41:47.037828 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":152,"threads_started":1}
I20260925 22:41:47.040793 12113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 505f452f86ed42c09303eb68bc7f54ce: No bootstrap required, opened a new log
I20260925 22:41:47.044670 12113 raft_consensus.cc:348] T 00000000000000000000000000000000 P 505f452f86ed42c09303eb68bc7f54ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "505f452f86ed42c09303eb68bc7f54ce" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33325 } }
I20260925 22:41:47.044917 12113 raft_consensus.cc:374] T 00000000000000000000000000000000 P 505f452f86ed42c09303eb68bc7f54ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:47.044961 12113 raft_consensus.cc:729] T 00000000000000000000000000000000 P 505f452f86ed42c09303eb68bc7f54ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 505f452f86ed42c09303eb68bc7f54ce, State: Initialized, Role: FOLLOWER
I20260925 22:41:47.045347 12113 consensus_queue.cc:260] T 00000000000000000000000000000000 P 505f452f86ed42c09303eb68bc7f54ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "505f452f86ed42c09303eb68bc7f54ce" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33325 } }
I20260925 22:41:47.045465 12113 raft_consensus.cc:388] T 00000000000000000000000000000000 P 505f452f86ed42c09303eb68bc7f54ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:41:47.045508 12113 raft_consensus.cc:482] T 00000000000000000000000000000000 P 505f452f86ed42c09303eb68bc7f54ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:41:47.045584 12113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 505f452f86ed42c09303eb68bc7f54ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:47.046535 12113 raft_consensus.cc:504] T 00000000000000000000000000000000 P 505f452f86ed42c09303eb68bc7f54ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "505f452f86ed42c09303eb68bc7f54ce" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33325 } }
I20260925 22:41:47.046854 12113 leader_election.cc:302] T 00000000000000000000000000000000 P 505f452f86ed42c09303eb68bc7f54ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 505f452f86ed42c09303eb68bc7f54ce; no voters:
I20260925 22:41:47.047223 12113 leader_election.cc:288] T 00000000000000000000000000000000 P 505f452f86ed42c09303eb68bc7f54ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:41:47.047386 12118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 505f452f86ed42c09303eb68bc7f54ce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:41:47.047670 12118 raft_consensus.cc:686] T 00000000000000000000000000000000 P 505f452f86ed42c09303eb68bc7f54ce [term 1 LEADER]: Becoming Leader. State: Replica: 505f452f86ed42c09303eb68bc7f54ce, State: Running, Role: LEADER
I20260925 22:41:47.048017 12118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 505f452f86ed42c09303eb68bc7f54ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "505f452f86ed42c09303eb68bc7f54ce" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33325 } }
I20260925 22:41:47.048615 12113 sys_catalog.cc:564] T 00000000000000000000000000000000 P 505f452f86ed42c09303eb68bc7f54ce [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:41:47.049845 12119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 505f452f86ed42c09303eb68bc7f54ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "505f452f86ed42c09303eb68bc7f54ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "505f452f86ed42c09303eb68bc7f54ce" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33325 } } }
I20260925 22:41:47.049880 12120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 505f452f86ed42c09303eb68bc7f54ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 505f452f86ed42c09303eb68bc7f54ce. Latest consensus state: current_term: 1 leader_uuid: "505f452f86ed42c09303eb68bc7f54ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "505f452f86ed42c09303eb68bc7f54ce" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33325 } } }
I20260925 22:41:47.049969 12119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 505f452f86ed42c09303eb68bc7f54ce [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:47.050000 12120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 505f452f86ed42c09303eb68bc7f54ce [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:47.050424 12127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:41:47.053913 12127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:41:47.060674 12127 catalog_manager.cc:1347] Generated new cluster ID: 5ecd7bb9c3a24af3acc11ac697fe0770
I20260925 22:41:47.060781 12127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:41:47.090145 12127 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:41:47.090536 12127 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:41:47.097496 12127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 505f452f86ed42c09303eb68bc7f54ce: Generated new TSK 0
I20260925 22:41:47.098448 12127 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:41:47.103308 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:0
--local_ip_for_outbound_sockets=127.2.74.193
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:33325
--builtin_ntp_servers=127.2.74.212:34375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 22:41:47.224258 12137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:47.224624 12137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:47.224705 12137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:47.229321 12137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:47.229425 12137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:47.229524 12137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:41:47.234678 12137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:34375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:33325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:47.235957 12137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:47.237450 12137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:47.247957 12143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:47.247957 12142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:47.247959 12145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:47.248015 12137 server_base.cc:1034] running on GCE node
I20260925 22:41:47.248734 12137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:47.249608 12137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:47.250825 12137 hybrid_clock.cc:648] HybridClock initialized: now 1790376107250778 us; error 72 us; skew 500 ppm
I20260925 22:41:47.252916 12137 webserver.cc:460] Webserver started at http://127.2.74.193:42891/ using document root <none> and password file <none>
I20260925 22:41:47.253575 12137 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:47.253681 12137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:47.253947 12137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:47.256296 12137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/data/instance:
uuid: "d7abcaa8e378434480e5f6be9997300a"
format_stamp: "Formatted at 2026-09-25 22:41:47 on dist-test-slave-csnl"
server_key: "74f83971a1ffaf326d10d7109fd9d7f5"
server_key_iv: "da3ca64ebaa470e2375fe83a2476fa6b"
server_key_version: "encryptionkey@0"
I20260925 22:41:47.256932 12137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance:
uuid: "d7abcaa8e378434480e5f6be9997300a"
format_stamp: "Formatted at 2026-09-25 22:41:47 on dist-test-slave-csnl"
server_key: "74f83971a1ffaf326d10d7109fd9d7f5"
server_key_iv: "da3ca64ebaa470e2375fe83a2476fa6b"
server_key_version: "encryptionkey@0"
I20260925 22:41:47.261070 12137 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:41:47.264038 12151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:47.265413 12137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:41:47.265588 12137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "d7abcaa8e378434480e5f6be9997300a"
format_stamp: "Formatted at 2026-09-25 22:41:47 on dist-test-slave-csnl"
server_key: "74f83971a1ffaf326d10d7109fd9d7f5"
server_key_iv: "da3ca64ebaa470e2375fe83a2476fa6b"
server_key_version: "encryptionkey@0"
I20260925 22:41:47.265735 12137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:47.270743 12137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:47.271538 12137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:47.271754 12137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:47.272343 12137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:47.273353 12137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:47.273428 12137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:47.273497 12137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:47.273550 12137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:47.283039 12137 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:46393
I20260925 22:41:47.283085 12264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:46393 every 8 connection(s)
I20260925 22:41:47.284050 12137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:41:47.290441 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 12137
I20260925 22:41:47.290616 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance
I20260925 22:41:47.290776 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:47.286051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:44265 (local address 127.2.74.254:33325)
0925 22:41:47.286235 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:47.286240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:47.287495 (+ 1255us) server_negotiation.cc:407] Connection header received
0925 22:41:47.288876 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:47.288879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:47.288921 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:47.289005 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:47.290271 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:47.290275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:47.290278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:41:47.290294 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:47.290320 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:41:47.290333 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:47.290458 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:47.290583 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 22:41:47.290637 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 22:41:47.291021 2347 external_mini_cluster.cc:1428] Setting key 5ed2135b8bd58518473afd3ab5f3fddf
I20260925 22:41:47.291867 12267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:47.286519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33325 (local address 127.2.74.193:44265)
0925 22:41:47.287305 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:47.287360 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:47.288602 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:47.289196 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:47.289204 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:47.289309 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:47.289926 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:41:47.289927 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:47.289979 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:41:47.290058 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:47.290436 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:47.290442 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:47.290447 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:41:47.290633 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 22:41:47.290956 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":123,"threads_started":1}
I20260925 22:41:47.293148 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:0
--local_ip_for_outbound_sockets=127.2.74.194
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:33325
--builtin_ntp_servers=127.2.74.212:34375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 22:41:47.293226 12265 heartbeater.cc:347] Connected to a master server at 127.2.74.254:33325
I20260925 22:41:47.293503 12265 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:47.294052 12265 heartbeater.cc:511] Master 127.2.74.254:33325 requested a full tablet report, sending...
I20260925 22:41:47.296146 12077 ts_manager.cc:194] Registered new tserver with Master: d7abcaa8e378434480e5f6be9997300a (127.2.74.193:46393)
W20260925 22:41:47.411477 12268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:47.411862 12268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:47.411947 12268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:47.416491 12268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:47.416600 12268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:47.416702 12268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:41:47.422051 12268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:34375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:33325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:47.423305 12268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:47.424800 12268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:47.435256 12273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:47.435372 12268 server_base.cc:1034] running on GCE node
W20260925 22:41:47.435245 12274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:47.435245 12276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:47.436125 12268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:47.436839 12268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:47.438047 12268 hybrid_clock.cc:648] HybridClock initialized: now 1790376107438029 us; error 41 us; skew 500 ppm
I20260925 22:41:47.440109 12268 webserver.cc:460] Webserver started at http://127.2.74.194:38143/ using document root <none> and password file <none>
I20260925 22:41:47.440733 12268 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:47.440799 12268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:47.441054 12268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:47.443300 12268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/data/instance:
uuid: "f9cc5f285f8a414d88f9e00ed1ce7def"
format_stamp: "Formatted at 2026-09-25 22:41:47 on dist-test-slave-csnl"
server_key: "b891a4fc4f3104f24f09d9a51c1f0a98"
server_key_iv: "82626cc775c5a7654a51d3f581927e6a"
server_key_version: "encryptionkey@0"
I20260925 22:41:47.443915 12268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance:
uuid: "f9cc5f285f8a414d88f9e00ed1ce7def"
format_stamp: "Formatted at 2026-09-25 22:41:47 on dist-test-slave-csnl"
server_key: "b891a4fc4f3104f24f09d9a51c1f0a98"
server_key_iv: "82626cc775c5a7654a51d3f581927e6a"
server_key_version: "encryptionkey@0"
I20260925 22:41:47.448117 12268 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:41:47.452186 12282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:47.454016 12268 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 22:41:47.454191 12268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "f9cc5f285f8a414d88f9e00ed1ce7def"
format_stamp: "Formatted at 2026-09-25 22:41:47 on dist-test-slave-csnl"
server_key: "b891a4fc4f3104f24f09d9a51c1f0a98"
server_key_iv: "82626cc775c5a7654a51d3f581927e6a"
server_key_version: "encryptionkey@0"
I20260925 22:41:47.454353 12268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:47.460641 12268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:47.461376 12268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:47.461587 12268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:47.462203 12268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:47.463292 12268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:47.463363 12268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:47.463454 12268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:47.463501 12268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:47.472604 12268 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:39579
I20260925 22:41:47.472656 12395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:39579 every 8 connection(s)
I20260925 22:41:47.473631 12268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:41:47.480154 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:47.475636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:40423 (local address 127.2.74.254:33325)
0925 22:41:47.475784 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:47.475788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:47.476930 (+ 1142us) server_negotiation.cc:407] Connection header received
0925 22:41:47.478327 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:47.478331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:47.478380 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:47.478463 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:47.479651 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:47.479655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:47.479658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:41:47.479671 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:47.479694 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:41:47.479710 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:47.479825 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:47.479957 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 22:41:47.480006 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:41:47.480630 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 12268
I20260925 22:41:47.480753 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance
I20260925 22:41:47.481079 2347 external_mini_cluster.cc:1428] Setting key 92bb8ed6651b2ed86523f38f363520b2
I20260925 22:41:47.481369 12398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:47.476069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33325 (local address 127.2.74.194:40423)
0925 22:41:47.476762 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:47.476806 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:47.478076 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:47.478658 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:47.478666 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:47.478764 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:47.479375 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:41:47.479376 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:47.479419 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:41:47.479492 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:47.479797 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:47.479806 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:47.479812 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:41:47.480008 (+ 196us) client_negotiation.cc:240] Negotiation successful
0925 22:41:47.480272 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":131,"threads_started":1}
I20260925 22:41:47.482545 12396 heartbeater.cc:347] Connected to a master server at 127.2.74.254:33325
I20260925 22:41:47.482815 12396 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:47.483358 12396 heartbeater.cc:511] Master 127.2.74.254:33325 requested a full tablet report, sending...
I20260925 22:41:47.483395 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:0
--local_ip_for_outbound_sockets=127.2.74.195
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:33325
--builtin_ntp_servers=127.2.74.212:34375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 22:41:47.484738 12077 ts_manager.cc:194] Registered new tserver with Master: f9cc5f285f8a414d88f9e00ed1ce7def (127.2.74.194:39579)
W20260925 22:41:47.602749 12399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:47.603158 12399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:47.603256 12399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:47.607679 12399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:47.607757 12399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:47.607822 12399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:41:47.613118 12399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:34375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:33325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:47.614478 12399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:47.616051 12399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:47.626367 12404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:47.626475 12405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:47.626367 12407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:47.626433 12399 server_base.cc:1034] running on GCE node
I20260925 22:41:47.628125 12399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:47.628854 12399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:47.630020 12399 hybrid_clock.cc:648] HybridClock initialized: now 1790376107630005 us; error 31 us; skew 500 ppm
I20260925 22:41:47.632076 12399 webserver.cc:460] Webserver started at http://127.2.74.195:36693/ using document root <none> and password file <none>
I20260925 22:41:47.632684 12399 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:47.632750 12399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:47.632987 12399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:47.635210 12399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/data/instance:
uuid: "cb50b3552e68425d9c3084c1e47a6145"
format_stamp: "Formatted at 2026-09-25 22:41:47 on dist-test-slave-csnl"
server_key: "de2504bf8b6aee8212ac2dcd98cc58a8"
server_key_iv: "0de255ee9e807880909bf7619a76aa75"
server_key_version: "encryptionkey@0"
I20260925 22:41:47.635797 12399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance:
uuid: "cb50b3552e68425d9c3084c1e47a6145"
format_stamp: "Formatted at 2026-09-25 22:41:47 on dist-test-slave-csnl"
server_key: "de2504bf8b6aee8212ac2dcd98cc58a8"
server_key_iv: "0de255ee9e807880909bf7619a76aa75"
server_key_version: "encryptionkey@0"
I20260925 22:41:47.639739 12399 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:41:47.642575 12413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:47.644076 12399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:41:47.644218 12399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "cb50b3552e68425d9c3084c1e47a6145"
format_stamp: "Formatted at 2026-09-25 22:41:47 on dist-test-slave-csnl"
server_key: "de2504bf8b6aee8212ac2dcd98cc58a8"
server_key_iv: "0de255ee9e807880909bf7619a76aa75"
server_key_version: "encryptionkey@0"
I20260925 22:41:47.644362 12399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:47.649348 12399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:47.650080 12399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:47.650278 12399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:47.650848 12399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:47.651839 12399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:47.651908 12399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:47.651979 12399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:47.652032 12399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:47.661026 12399 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:45401
I20260925 22:41:47.661087 12526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:45401 every 8 connection(s)
I20260925 22:41:47.662065 12399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:41:47.668648 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:47.664101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:56753 (local address 127.2.74.254:33325)
0925 22:41:47.664257 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:47.664261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:47.665471 (+ 1210us) server_negotiation.cc:407] Connection header received
0925 22:41:47.667038 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:47.667042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:47.667084 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:47.667163 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:47.668262 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:47.668265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:47.668267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:41:47.668279 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:47.668296 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:41:47.668304 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:47.668383 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:47.668502 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 22:41:47.668538 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:41:47.669759 12529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:47.664607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33325 (local address 127.2.74.195:56753)
0925 22:41:47.665313 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:47.665359 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:47.666756 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:47.667337 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:47.667344 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:47.667444 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:47.667999 (+ 555us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:41:47.668000 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:47.668042 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:41:47.668116 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:47.668380 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:47.668386 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:47.668390 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 22:41:47.668527 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 22:41:47.668917 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":162,"threads_started":1}
I20260925 22:41:47.670306 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 12399
I20260925 22:41:47.670444 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance
I20260925 22:41:47.670775 2347 external_mini_cluster.cc:1428] Setting key f40f2e95a140c4a8388607e7b2e67282
I20260925 22:41:47.671000 12527 heartbeater.cc:347] Connected to a master server at 127.2.74.254:33325
I20260925 22:41:47.671277 12527 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:47.671779 12527 heartbeater.cc:511] Master 127.2.74.254:33325 requested a full tablet report, sending...
I20260925 22:41:47.672942 12077 ts_manager.cc:194] Registered new tserver with Master: cb50b3552e68425d9c3084c1e47a6145 (127.2.74.195:45401)
I20260925 22:41:47.684991 2347 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 22:41:47.688012 12116 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:41:47.687193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59742 (local address 127.2.74.254:33325)
0925 22:41:47.687307 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:47.687311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:47.687477 (+ 166us) server_negotiation.cc:407] Connection header received
0925 22:41:47.687604 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:47.687612 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:47.687645 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:47.687706 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:47.687903 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59742: BlockingRecv error: recv got EOF from 127.0.0.1:59742 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 22:41:47.688248 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 12137
I20260925 22:41:47.693558 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 12268
I20260925 22:41:47.699175 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 12399
I20260925 22:41:47.704845 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 12046
2026-09-25T22:41:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (879 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:41:47 dist-test-slave-csnl krb5kdc[12537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:41:47 dist-test-slave-csnl krb5kdc[12537](info): set up 2 sockets
Sep 25 22:41:47 dist-test-slave-csnl krb5kdc[12537](info): commencing operation
krb5kdc: starting...
W20260925 22:41:48.768150 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:41:48 dist-test-slave-csnl krb5kdc[12537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:41:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:41:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:41:48.898510 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:43001
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:34909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:43001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:49.014469 12550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:49.014844 12550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:49.014941 12550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:49.019366 12550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:41:49.019507 12550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:49.019544 12550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:49.019573 12550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:41:49.019629 12550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:41:49.025657 12550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:34909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:43001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:43001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:49.026909 12550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:49.028052 12550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:49.037587 12558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:49.037578 12555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:49.037578 12556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:49.037637 12550 server_base.cc:1034] running on GCE node
I20260925 22:41:49.038415 12550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:49.039772 12550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:49.040943 12550 hybrid_clock.cc:648] HybridClock initialized: now 1790376109040928 us; error 37 us; skew 500 ppm
Sep 25 22:41:49 dist-test-slave-csnl krb5kdc[12537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376109, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:49.050334 12550 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:41:49.051560 12550 webserver.cc:460] Webserver started at http://127.2.74.254:43573/ using document root <none> and password file <none>
I20260925 22:41:49.052205 12550 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:49.052309 12550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:49.052567 12550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:49.054836 12550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "f32c24fc95d543a1a39676f9b4fc5171"
format_stamp: "Formatted at 2026-09-25 22:41:49 on dist-test-slave-csnl"
server_key: "59b3bcc6fe657213a6ed4e83dd88642d"
server_key_iv: "733bf2e0865ffb60f86fce5a972d8744"
server_key_version: "encryptionkey@0"
I20260925 22:41:49.055542 12550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "f32c24fc95d543a1a39676f9b4fc5171"
format_stamp: "Formatted at 2026-09-25 22:41:49 on dist-test-slave-csnl"
server_key: "59b3bcc6fe657213a6ed4e83dd88642d"
server_key_iv: "733bf2e0865ffb60f86fce5a972d8744"
server_key_version: "encryptionkey@0"
I20260925 22:41:49.059614 12550 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:41:49.062314 12565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:49.063702 12550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:41:49.063863 12550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "f32c24fc95d543a1a39676f9b4fc5171"
format_stamp: "Formatted at 2026-09-25 22:41:49 on dist-test-slave-csnl"
server_key: "59b3bcc6fe657213a6ed4e83dd88642d"
server_key_iv: "733bf2e0865ffb60f86fce5a972d8744"
server_key_version: "encryptionkey@0"
I20260925 22:41:49.064043 12550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:49.076902 12550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:49.080479 12550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:49.080700 12550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:49.088308 12550 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:43001
I20260925 22:41:49.088598 12617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:43001 every 8 connection(s)
I20260925 22:41:49.089407 12550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:41:49.092401 12618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:49.095640 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 12550
I20260925 22:41:49.095779 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:41:49.096077 2347 external_mini_cluster.cc:1428] Setting key 739996ecd44f58398cc764a9f7a24e07
I20260925 22:41:49.099090 12618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f32c24fc95d543a1a39676f9b4fc5171: Bootstrap starting.
I20260925 22:41:49.102196 12618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f32c24fc95d543a1a39676f9b4fc5171: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:49.103183 12618 log.cc:824] T 00000000000000000000000000000000 P f32c24fc95d543a1a39676f9b4fc5171: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:49.105479 12618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f32c24fc95d543a1a39676f9b4fc5171: No bootstrap required, opened a new log
Sep 25 22:41:49 dist-test-slave-csnl krb5kdc[12537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:41:49.110391 12618 raft_consensus.cc:348] T 00000000000000000000000000000000 P f32c24fc95d543a1a39676f9b4fc5171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32c24fc95d543a1a39676f9b4fc5171" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 43001 } }
I20260925 22:41:49.110751 12618 raft_consensus.cc:374] T 00000000000000000000000000000000 P f32c24fc95d543a1a39676f9b4fc5171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:49.110841 12618 raft_consensus.cc:729] T 00000000000000000000000000000000 P f32c24fc95d543a1a39676f9b4fc5171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f32c24fc95d543a1a39676f9b4fc5171, State: Initialized, Role: FOLLOWER
I20260925 22:41:49.111454 12618 consensus_queue.cc:260] T 00000000000000000000000000000000 P f32c24fc95d543a1a39676f9b4fc5171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32c24fc95d543a1a39676f9b4fc5171" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 43001 } }
I20260925 22:41:49.111627 12618 raft_consensus.cc:388] T 00000000000000000000000000000000 P f32c24fc95d543a1a39676f9b4fc5171 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:41:49.111702 12618 raft_consensus.cc:482] T 00000000000000000000000000000000 P f32c24fc95d543a1a39676f9b4fc5171 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:41:49.111835 12618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f32c24fc95d543a1a39676f9b4fc5171 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:49.113160 12618 raft_consensus.cc:504] T 00000000000000000000000000000000 P f32c24fc95d543a1a39676f9b4fc5171 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32c24fc95d543a1a39676f9b4fc5171" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 43001 } }
I20260925 22:41:49.113551 12618 leader_election.cc:302] T 00000000000000000000000000000000 P f32c24fc95d543a1a39676f9b4fc5171 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f32c24fc95d543a1a39676f9b4fc5171; no voters:
I20260925 22:41:49.113731 12621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:49.097701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44922 (local address 127.2.74.254:43001)
0925 22:41:49.098167 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:49.098188 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:49.098236 (+ 48us) server_negotiation.cc:407] Connection header received
0925 22:41:49.099174 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:49.099203 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:49.099686 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:49.100094 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:49.100827 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:49.102357 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:49.103391 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:49.104006 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:49.104221 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:49.106780 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:49.106787 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:49.106808 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:49.106846 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:49.109973 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:49.110545 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:49.110551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:49.110557 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:49.110707 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:49.111084 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:49.111087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:49.111089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:49.111654 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:49.111881 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:49.112266 (+ 385us) server_negotiation.cc:299] Negotiation successful
0925 22:41:49.112605 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":121,"threads_started":1}
I20260925 22:41:49.113922 12618 leader_election.cc:288] T 00000000000000000000000000000000 P f32c24fc95d543a1a39676f9b4fc5171 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:41:49.114161 12623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f32c24fc95d543a1a39676f9b4fc5171 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:41:49.114358 12623 raft_consensus.cc:686] T 00000000000000000000000000000000 P f32c24fc95d543a1a39676f9b4fc5171 [term 1 LEADER]: Becoming Leader. State: Replica: f32c24fc95d543a1a39676f9b4fc5171, State: Running, Role: LEADER
I20260925 22:41:49.114667 12623 consensus_queue.cc:237] T 00000000000000000000000000000000 P f32c24fc95d543a1a39676f9b4fc5171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32c24fc95d543a1a39676f9b4fc5171" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 43001 } }
I20260925 22:41:49.115232 12618 sys_catalog.cc:564] T 00000000000000000000000000000000 P f32c24fc95d543a1a39676f9b4fc5171 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:41:49.116066 12625 sys_catalog.cc:455] T 00000000000000000000000000000000 P f32c24fc95d543a1a39676f9b4fc5171 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f32c24fc95d543a1a39676f9b4fc5171. Latest consensus state: current_term: 1 leader_uuid: "f32c24fc95d543a1a39676f9b4fc5171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32c24fc95d543a1a39676f9b4fc5171" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 43001 } } }
I20260925 22:41:49.116218 12625 sys_catalog.cc:458] T 00000000000000000000000000000000 P f32c24fc95d543a1a39676f9b4fc5171 [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:49.116509 12624 sys_catalog.cc:455] T 00000000000000000000000000000000 P f32c24fc95d543a1a39676f9b4fc5171 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f32c24fc95d543a1a39676f9b4fc5171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32c24fc95d543a1a39676f9b4fc5171" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 43001 } } }
I20260925 22:41:49.116590 12624 sys_catalog.cc:458] T 00000000000000000000000000000000 P f32c24fc95d543a1a39676f9b4fc5171 [sys.catalog]: This master's current role is: LEADER
I20260925 22:41:49.117128 12632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:41:49.120635 12632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:41:49.127496 12632 catalog_manager.cc:1347] Generated new cluster ID: 0537bc2acc0845d2b20f36389213c70a
I20260925 22:41:49.127602 12632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:41:49.149318 12632 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:41:49.150403 12632 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:41:49.158371 12632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f32c24fc95d543a1a39676f9b4fc5171: Generated new TSK 0
I20260925 22:41:49.159456 12632 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
I20260925 22:41:49.200255 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:0
--local_ip_for_outbound_sockets=127.2.74.193
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:43001
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:34909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:49.325922 12644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:49.326303 12644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:49.326355 12644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:49.330744 12644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:49.330842 12644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:49.330910 12644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:41:49.336879 12644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:34909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:43001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:49.338027 12644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:49.339090 12644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:49.349560 12649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:49.349651 12644 server_base.cc:1034] running on GCE node
W20260925 22:41:49.349571 12652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:49.349575 12650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:49.350430 12644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:49.351225 12644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:49.352440 12644 hybrid_clock.cc:648] HybridClock initialized: now 1790376109352423 us; error 32 us; skew 500 ppm
Sep 25 22:41:49 dist-test-slave-csnl krb5kdc[12537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376109, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:49.355899 12644 init.cc:377] Logged in from keytab as kudu/127.2.74.193@KRBTEST.COM (short username kudu)
I20260925 22:41:49.356973 12644 webserver.cc:460] Webserver started at http://127.2.74.193:39581/ using document root <none> and password file <none>
I20260925 22:41:49.357589 12644 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:49.357659 12644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:49.357861 12644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:49.360298 12644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/data/instance:
uuid: "50530183307349a9b135d15862167300"
format_stamp: "Formatted at 2026-09-25 22:41:49 on dist-test-slave-csnl"
server_key: "995344d8a32be90ef0a3bff2c5574ce2"
server_key_iv: "d78f6ef8e886e146da88d338fdb5ad72"
server_key_version: "encryptionkey@0"
I20260925 22:41:49.360918 12644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance:
uuid: "50530183307349a9b135d15862167300"
format_stamp: "Formatted at 2026-09-25 22:41:49 on dist-test-slave-csnl"
server_key: "995344d8a32be90ef0a3bff2c5574ce2"
server_key_iv: "d78f6ef8e886e146da88d338fdb5ad72"
server_key_version: "encryptionkey@0"
I20260925 22:41:49.365351 12644 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:41:49.368310 12659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:49.369751 12644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:41:49.369903 12644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "50530183307349a9b135d15862167300"
format_stamp: "Formatted at 2026-09-25 22:41:49 on dist-test-slave-csnl"
server_key: "995344d8a32be90ef0a3bff2c5574ce2"
server_key_iv: "d78f6ef8e886e146da88d338fdb5ad72"
server_key_version: "encryptionkey@0"
I20260925 22:41:49.370031 12644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:49.396371 12644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:49.399904 12644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:49.400131 12644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:49.400761 12644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:49.401604 12644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:49.401650 12644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:49.401718 12644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:49.401764 12644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:49.410907 12644 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:35729
I20260925 22:41:49.410928 12772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:35729 every 8 connection(s)
I20260925 22:41:49.411921 12644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:41:49.418179 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 12644
I20260925 22:41:49.418286 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance
I20260925 22:41:49.418577 2347 external_mini_cluster.cc:1428] Setting key b3796ef28901c324da8995d8ef7d66c8
Sep 25 22:41:49 dist-test-slave-csnl krb5kdc[12537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376109, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:41:49.428638 12621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:49.413833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:52539 (local address 127.2.74.254:43001)
0925 22:41:49.413969 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:49.413973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:49.414814 (+ 841us) server_negotiation.cc:407] Connection header received
0925 22:41:49.416430 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:49.416434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:49.416494 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:49.416569 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:49.417925 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:49.418678 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:49.420247 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:49.420475 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:49.420565 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:49.424241 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:49.424244 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:49.424246 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:49.424274 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:49.425934 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:49.426741 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:49.426745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:49.426746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:49.426850 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:49.427318 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:49.427322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:49.427324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:49.427539 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:49.427665 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:49.428281 (+ 616us) server_negotiation.cc:299] Negotiation successful
0925 22:41:49.428440 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.2.74.194@KRBTEST.COM; defaulting to no policy
I20260925 22:41:49.429600 12775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:49.414223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:43001 (local address 127.2.74.193:52539)
0925 22:41:49.414655 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:49.414699 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:49.416188 (+ 1489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:49.416746 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:49.416753 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:49.417242 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:49.417751 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:49.417766 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:49.418823 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:49.418828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:49.420122 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:49.420136 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:49.420650 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:49.420658 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:49.420928 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:49.421678 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:49.421706 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:49.424010 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:49.426100 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:49.426107 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:49.426153 (+ 46us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:49.426588 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:49.426991 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:49.426995 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:49.426998 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:49.427174 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:49.427666 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:49.427675 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:49.427987 (+ 312us) client_negotiation.cc:769] Sending connection context
0925 22:41:49.428210 (+ 223us) client_negotiation.cc:240] Negotiation successful
0925 22:41:49.428604 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":108,"threads_started":1}
I20260925 22:41:49.431087 12773 heartbeater.cc:347] Connected to a master server at 127.2.74.254:43001
I20260925 22:41:49.431367 12773 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:49.431945 12773 heartbeater.cc:511] Master 127.2.74.254:43001 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.194@KRBTEST.COM" created.
I20260925 22:41:49.433739 12582 ts_manager.cc:194] Registered new tserver with Master: 50530183307349a9b135d15862167300 (127.2.74.193:35729)
I20260925 22:41:49.435068 12582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.193@KRBTEST.COM'} at 127.2.74.193:52539
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
I20260925 22:41:49.450325 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:0
--local_ip_for_outbound_sockets=127.2.74.194
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:43001
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:34909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:49.567713 12778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:49.568030 12778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:49.568125 12778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:49.572420 12778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:49.572530 12778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:49.572630 12778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:41:49.578336 12778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:34909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:43001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:49.579579 12778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:49.580835 12778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:49.591244 12783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:49.591259 12784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:49.591332 12778 server_base.cc:1034] running on GCE node
W20260925 22:41:49.591244 12786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:49.591892 12778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:49.592603 12778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:49.593788 12778 hybrid_clock.cc:648] HybridClock initialized: now 1790376109593744 us; error 64 us; skew 500 ppm
Sep 25 22:41:49 dist-test-slave-csnl krb5kdc[12537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376109, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:49.597010 12778 init.cc:377] Logged in from keytab as kudu/127.2.74.194@KRBTEST.COM (short username kudu)
I20260925 22:41:49.598045 12778 webserver.cc:460] Webserver started at http://127.2.74.194:46237/ using document root <none> and password file <none>
I20260925 22:41:49.598683 12778 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:49.598789 12778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:49.599098 12778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:49.601372 12778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/data/instance:
uuid: "06b8cf35c542444a891be6483830f742"
format_stamp: "Formatted at 2026-09-25 22:41:49 on dist-test-slave-csnl"
server_key: "d9ba5ebfb28913af339355ef521af6b5"
server_key_iv: "eaf5abde4c04a3bde79fae09b604697f"
server_key_version: "encryptionkey@0"
I20260925 22:41:49.602053 12778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance:
uuid: "06b8cf35c542444a891be6483830f742"
format_stamp: "Formatted at 2026-09-25 22:41:49 on dist-test-slave-csnl"
server_key: "d9ba5ebfb28913af339355ef521af6b5"
server_key_iv: "eaf5abde4c04a3bde79fae09b604697f"
server_key_version: "encryptionkey@0"
I20260925 22:41:49.606439 12778 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:41:49.609381 12793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:49.610780 12778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:41:49.611035 12778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "06b8cf35c542444a891be6483830f742"
format_stamp: "Formatted at 2026-09-25 22:41:49 on dist-test-slave-csnl"
server_key: "d9ba5ebfb28913af339355ef521af6b5"
server_key_iv: "eaf5abde4c04a3bde79fae09b604697f"
server_key_version: "encryptionkey@0"
I20260925 22:41:49.611181 12778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:49.637737 12778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:49.640857 12778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:49.641074 12778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:49.641686 12778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:49.642571 12778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:49.642652 12778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:49.642725 12778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:49.642777 12778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:49.652325 12778 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:37317
I20260925 22:41:49.652354 12906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:37317 every 8 connection(s)
I20260925 22:41:49.653260 12778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:41:49.657646 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 12778
I20260925 22:41:49.657771 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance
I20260925 22:41:49.658093 2347 external_mini_cluster.cc:1428] Setting key f390749598a3398519b97fc57830dc9f
Sep 25 22:41:49 dist-test-slave-csnl krb5kdc[12537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376109, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.74.195@KRBTEST.COM; defaulting to no policy
I20260925 22:41:49.668259 12621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:49.655203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:38269 (local address 127.2.74.254:43001)
0925 22:41:49.655370 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:49.655374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:49.656274 (+ 900us) server_negotiation.cc:407] Connection header received
0925 22:41:49.657691 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:49.657695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:49.657749 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:49.657858 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:49.659328 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:49.659942 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:49.661219 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:49.661486 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:49.661574 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:49.664512 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:49.664516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:49.664518 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:49.664541 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:49.665983 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:49.666530 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:49.666539 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:49.666540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:49.666614 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:49.666988 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:49.666991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:49.666993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:49.667245 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:49.667358 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:49.667967 (+ 609us) server_negotiation.cc:299] Negotiation successful
0925 22:41:49.668094 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 22:41:49.669242 12909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:49.655598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:43001 (local address 127.2.74.194:38269)
0925 22:41:49.656097 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:49.656146 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:49.657475 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:49.658106 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:49.658114 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:49.658561 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:49.659186 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:49.659201 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:49.660079 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:49.660083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:49.661086 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:49.661100 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:49.661649 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:49.661654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:49.661904 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:49.662556 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:49.662583 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:49.664313 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:49.666108 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:49.666114 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:49.666132 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:49.666415 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:49.666706 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:49.666711 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:49.666713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:49.666866 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:49.667405 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:49.667417 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:49.667708 (+ 291us) client_negotiation.cc:769] Sending connection context
0925 22:41:49.667903 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 22:41:49.668274 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.195@KRBTEST.COM" created.
I20260925 22:41:49.670550 12907 heartbeater.cc:347] Connected to a master server at 127.2.74.254:43001
I20260925 22:41:49.670816 12907 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:49.671360 12907 heartbeater.cc:511] Master 127.2.74.254:43001 requested a full tablet report, sending...
I20260925 22:41:49.672434 12582 ts_manager.cc:194] Registered new tserver with Master: 06b8cf35c542444a891be6483830f742 (127.2.74.194:37317)
I20260925 22:41:49.673024 12582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.194@KRBTEST.COM'} at 127.2.74.194:38269
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
I20260925 22:41:49.686544 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:0
--local_ip_for_outbound_sockets=127.2.74.195
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:43001
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:34909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:41:49.807327 12912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:41:49.807647 12912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:41:49.807742 12912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:41:49.812114 12912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:41:49.812206 12912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:41:49.812305 12912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:41:49.818462 12912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:34909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:43001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:41:49.819723 12912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:41:49.820845 12912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:41:49.831187 12918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:49.831424 12912 server_base.cc:1034] running on GCE node
W20260925 22:41:49.831187 12917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:41:49.831804 12920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:41:49.832284 12912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:41:49.832993 12912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:41:49.834160 12912 hybrid_clock.cc:648] HybridClock initialized: now 1790376109834126 us; error 55 us; skew 500 ppm
Sep 25 22:41:49 dist-test-slave-csnl krb5kdc[12537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376109, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:41:49.837013 12912 init.cc:377] Logged in from keytab as kudu/127.2.74.195@KRBTEST.COM (short username kudu)
I20260925 22:41:49.837879 12912 webserver.cc:460] Webserver started at http://127.2.74.195:46509/ using document root <none> and password file <none>
I20260925 22:41:49.838414 12912 fs_manager.cc:359] Metadata directory not provided
I20260925 22:41:49.838474 12912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:41:49.838636 12912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:41:49.840724 12912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/data/instance:
uuid: "5df199e5ba78424cbbf3a5a87e61e960"
format_stamp: "Formatted at 2026-09-25 22:41:49 on dist-test-slave-csnl"
server_key: "ce79b1e460f2cdc896339c4a786d631f"
server_key_iv: "1641c05018f9cdf1a5a026b84858dbd2"
server_key_version: "encryptionkey@0"
I20260925 22:41:49.841253 12912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance:
uuid: "5df199e5ba78424cbbf3a5a87e61e960"
format_stamp: "Formatted at 2026-09-25 22:41:49 on dist-test-slave-csnl"
server_key: "ce79b1e460f2cdc896339c4a786d631f"
server_key_iv: "1641c05018f9cdf1a5a026b84858dbd2"
server_key_version: "encryptionkey@0"
I20260925 22:41:49.845072 12912 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 22:41:49.847651 12927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:49.849018 12912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:41:49.849139 12912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "5df199e5ba78424cbbf3a5a87e61e960"
format_stamp: "Formatted at 2026-09-25 22:41:49 on dist-test-slave-csnl"
server_key: "ce79b1e460f2cdc896339c4a786d631f"
server_key_iv: "1641c05018f9cdf1a5a026b84858dbd2"
server_key_version: "encryptionkey@0"
I20260925 22:41:49.849247 12912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:41:49.864991 12912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:41:49.868780 12912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:41:49.868999 12912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:41:49.869619 12912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:41:49.870456 12912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:41:49.870507 12912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:49.870548 12912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:41:49.870564 12912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:41:49.880198 12912 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:38093
I20260925 22:41:49.880236 13040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:38093 every 8 connection(s)
I20260925 22:41:49.881172 12912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:41:49.883945 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 12912
I20260925 22:41:49.884087 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance
I20260925 22:41:49.884440 2347 external_mini_cluster.cc:1428] Setting key e4539bce4ad8e7e2bc19b66052474935
Sep 25 22:41:49 dist-test-slave-csnl krb5kdc[12537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376109, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:41:49.896410 12621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:49.883201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:59925 (local address 127.2.74.254:43001)
0925 22:41:49.883349 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:49.883353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:49.884410 (+ 1057us) server_negotiation.cc:407] Connection header received
0925 22:41:49.886277 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:49.886281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:49.886340 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:49.886439 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:49.887870 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:49.888462 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:49.889669 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:49.889918 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:49.890009 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:49.892784 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:49.892813 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:49.892815 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:49.892843 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:49.894200 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:49.894783 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:49.894786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:49.894787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:49.894882 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:49.895294 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:49.895297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:49.895299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:49.895486 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:49.895577 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:49.896169 (+ 592us) server_negotiation.cc:299] Negotiation successful
0925 22:41:49.896285 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:41:49.897426 13043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:49.883609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:43001 (local address 127.2.74.195:59925)
0925 22:41:49.884180 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:49.884251 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:49.886010 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:49.886582 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:49.886590 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:49.887167 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:41:49.887714 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:49.887729 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:49.888597 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:49.888600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:49.889539 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:49.889549 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:49.890077 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:49.890080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:49.890224 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:49.891049 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:41:49.891070 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:41:49.892584 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:41:49.894344 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:49.894350 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:49.894365 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:49.894678 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:49.895049 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:41:49.895051 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:41:49.895052 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:41:49.895198 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:41:49.895605 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:41:49.895610 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:41:49.895912 (+ 302us) client_negotiation.cc:769] Sending connection context
0925 22:41:49.896080 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 22:41:49.896363 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":136,"threads_started":1}
I20260925 22:41:49.898628 13041 heartbeater.cc:347] Connected to a master server at 127.2.74.254:43001
I20260925 22:41:49.898895 13041 heartbeater.cc:464] Registering TS with master...
I20260925 22:41:49.899413 13041 heartbeater.cc:511] Master 127.2.74.254:43001 requested a full tablet report, sending...
I20260925 22:41:49.900377 12582 ts_manager.cc:194] Registered new tserver with Master: 5df199e5ba78424cbbf3a5a87e61e960 (127.2.74.195:38093)
I20260925 22:41:49.900947 12582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.195@KRBTEST.COM'} at 127.2.74.195:59925
I20260925 22:41:49.911024 2347 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:41:49.921206 12621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:49.913310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44936 (local address 127.2.74.254:43001)
0925 22:41:49.913451 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:49.913457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:49.913538 (+ 81us) server_negotiation.cc:407] Connection header received
0925 22:41:49.913705 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:49.913708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:49.913759 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:49.913848 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:41:49.914591 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:49.915326 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:49.916353 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:49.916579 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:49.916687 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:49.918152 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:41:49.918156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:41:49.918159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:41:49.918193 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:41:49.919500 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:49.919923 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:49.919927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:49.919928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:49.920010 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:41:49.920279 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:41:49.920284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:41:49.920286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:41:49.920518 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:41:49.920658 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:49.920918 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 22:41:49.921043 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:41:49.924446 12582 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:41:49.927523 12582 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:41:49.942365 13056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:49.936859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44950 (local address 127.2.74.193:35729)
0925 22:41:49.937221 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:49.937229 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:49.937422 (+ 193us) server_negotiation.cc:407] Connection header received
0925 22:41:49.938125 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:49.938132 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:49.938320 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:49.938469 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:49.938636 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:49.939644 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:49.941020 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:49.941632 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:49.941740 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:49.941925 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:49.942110 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 22:41:49.942210 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":92,"threads_started":1}
I20260925 22:41:49.942371 13054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:49.936447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:35729 (local address 127.0.0.1:44950)
0925 22:41:49.937206 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:49.937263 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:49.937411 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:49.938447 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:49.938450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:49.938466 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:49.938544 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:49.938551 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:49.939775 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:49.939778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:49.940929 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:49.940938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:49.941772 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:49.941776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:49.941906 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:49.941952 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 22:41:49.942068 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 22:41:49.942191 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"mutex_wait_us":4,"thread_start_us":108,"threads_started":1}
I20260925 22:41:49.942759 13053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:49.936235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58246 (local address 127.2.74.195:38093)
0925 22:41:49.936733 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:49.937191 (+ 458us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:49.937369 (+ 178us) server_negotiation.cc:407] Connection header received
0925 22:41:49.937556 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:49.937562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:49.937749 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:49.937908 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:49.938266 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:49.939254 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:49.940945 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:49.941759 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:49.941878 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:49.942028 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:49.942501 (+ 473us) server_negotiation.cc:299] Negotiation successful
0925 22:41:49.942603 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":179,"threads_started":1}
I20260925 22:41:49.942415 13052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:49.936065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:38093 (local address 127.0.0.1:58246)
0925 22:41:49.937207 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:49.937263 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:49.937411 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:49.937914 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:49.937920 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:49.937946 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:49.938102 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:49.938110 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:49.939404 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:49.939408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:49.940812 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:49.940822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:49.942090 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:49.942093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:49.942173 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:49.942192 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 22:41:49.942257 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 22:41:49.942311 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":977,"thread_start_us":110,"threads_started":1}
I20260925 22:41:49.943502 13055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:49.936800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:37317 (local address 127.0.0.1:38588)
0925 22:41:49.937512 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:49.937524 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:49.937615 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:49.938023 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:49.938026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:49.938043 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:49.938151 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:49.938159 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:49.940453 (+ 2294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:49.940459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:49.941431 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:49.941441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:49.942922 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:49.942926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:49.943193 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:49.943217 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:41:49.943283 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 22:41:49.943354 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":105,"threads_started":1}
I20260925 22:41:49.943691 13057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:49.936944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38588 (local address 127.2.74.194:37317)
0925 22:41:49.937373 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:49.937384 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:49.937616 (+ 232us) server_negotiation.cc:407] Connection header received
0925 22:41:49.937721 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:49.937726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:49.937886 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:49.938044 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:49.939420 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:49.940318 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:49.941575 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:49.942417 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:49.942669 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:49.943404 (+ 735us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:49.943464 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 22:41:49.943558 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":91,"threads_started":1}
I20260925 22:41:49.945710 12975 tablet_service.cc:1467] Processing CreateTablet for tablet 5614bacc4579446793ad8349372c58b0 (DEFAULT_TABLE table=test-table [id=e756eaaca93745dbaf8db64cbd6484ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:49.945714 12707 tablet_service.cc:1467] Processing CreateTablet for tablet 5614bacc4579446793ad8349372c58b0 (DEFAULT_TABLE table=test-table [id=e756eaaca93745dbaf8db64cbd6484ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:49.946782 12975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5614bacc4579446793ad8349372c58b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:49.946779 12707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5614bacc4579446793ad8349372c58b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:49.949151 12841 tablet_service.cc:1467] Processing CreateTablet for tablet 5614bacc4579446793ad8349372c58b0 (DEFAULT_TABLE table=test-table [id=e756eaaca93745dbaf8db64cbd6484ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:41:49.950266 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5614bacc4579446793ad8349372c58b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:41:49.953158 13058 tablet_bootstrap.cc:492] T 5614bacc4579446793ad8349372c58b0 P 5df199e5ba78424cbbf3a5a87e61e960: Bootstrap starting.
I20260925 22:41:49.954877 13059 tablet_bootstrap.cc:492] T 5614bacc4579446793ad8349372c58b0 P 50530183307349a9b135d15862167300: Bootstrap starting.
I20260925 22:41:49.956511 13058 tablet_bootstrap.cc:654] T 5614bacc4579446793ad8349372c58b0 P 5df199e5ba78424cbbf3a5a87e61e960: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:49.957283 13059 tablet_bootstrap.cc:654] T 5614bacc4579446793ad8349372c58b0 P 50530183307349a9b135d15862167300: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:49.957329 13058 log.cc:824] T 5614bacc4579446793ad8349372c58b0 P 5df199e5ba78424cbbf3a5a87e61e960: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:49.958103 13059 log.cc:824] T 5614bacc4579446793ad8349372c58b0 P 50530183307349a9b135d15862167300: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:49.959074 13061 tablet_bootstrap.cc:492] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742: Bootstrap starting.
I20260925 22:41:49.959694 13058 tablet_bootstrap.cc:492] T 5614bacc4579446793ad8349372c58b0 P 5df199e5ba78424cbbf3a5a87e61e960: No bootstrap required, opened a new log
I20260925 22:41:49.959923 13058 ts_tablet_manager.cc:1397] T 5614bacc4579446793ad8349372c58b0 P 5df199e5ba78424cbbf3a5a87e61e960: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:41:49.960628 13059 tablet_bootstrap.cc:492] T 5614bacc4579446793ad8349372c58b0 P 50530183307349a9b135d15862167300: No bootstrap required, opened a new log
I20260925 22:41:49.960814 13059 ts_tablet_manager.cc:1397] T 5614bacc4579446793ad8349372c58b0 P 50530183307349a9b135d15862167300: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:41:49.962033 13061 tablet_bootstrap.cc:654] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742: Neither blocks nor log segments found. Creating new log.
I20260925 22:41:49.963150 13061 log.cc:824] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742: Log is configured to *not* fsync() on all Append() calls
I20260925 22:41:49.963620 13058 raft_consensus.cc:348] T 5614bacc4579446793ad8349372c58b0 P 5df199e5ba78424cbbf3a5a87e61e960 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5df199e5ba78424cbbf3a5a87e61e960" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 38093 } } peers { permanent_uuid: "06b8cf35c542444a891be6483830f742" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37317 } } peers { permanent_uuid: "50530183307349a9b135d15862167300" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 35729 } }
I20260925 22:41:49.963936 13058 raft_consensus.cc:374] T 5614bacc4579446793ad8349372c58b0 P 5df199e5ba78424cbbf3a5a87e61e960 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:49.964002 13058 raft_consensus.cc:729] T 5614bacc4579446793ad8349372c58b0 P 5df199e5ba78424cbbf3a5a87e61e960 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5df199e5ba78424cbbf3a5a87e61e960, State: Initialized, Role: FOLLOWER
I20260925 22:41:49.964479 13058 consensus_queue.cc:260] T 5614bacc4579446793ad8349372c58b0 P 5df199e5ba78424cbbf3a5a87e61e960 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5df199e5ba78424cbbf3a5a87e61e960" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 38093 } } peers { permanent_uuid: "06b8cf35c542444a891be6483830f742" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37317 } } peers { permanent_uuid: "50530183307349a9b135d15862167300" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 35729 } }
I20260925 22:41:49.965209 13041 heartbeater.cc:503] Master 127.2.74.254:43001 was elected leader, sending a full tablet report...
I20260925 22:41:49.965601 13061 tablet_bootstrap.cc:492] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742: No bootstrap required, opened a new log
I20260925 22:41:49.965538 13059 raft_consensus.cc:348] T 5614bacc4579446793ad8349372c58b0 P 50530183307349a9b135d15862167300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5df199e5ba78424cbbf3a5a87e61e960" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 38093 } } peers { permanent_uuid: "06b8cf35c542444a891be6483830f742" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37317 } } peers { permanent_uuid: "50530183307349a9b135d15862167300" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 35729 } }
I20260925 22:41:49.965811 13059 raft_consensus.cc:374] T 5614bacc4579446793ad8349372c58b0 P 50530183307349a9b135d15862167300 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:49.965847 13061 ts_tablet_manager.cc:1397] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260925 22:41:49.965885 13059 raft_consensus.cc:729] T 5614bacc4579446793ad8349372c58b0 P 50530183307349a9b135d15862167300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50530183307349a9b135d15862167300, State: Initialized, Role: FOLLOWER
I20260925 22:41:49.966357 13059 consensus_queue.cc:260] T 5614bacc4579446793ad8349372c58b0 P 50530183307349a9b135d15862167300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5df199e5ba78424cbbf3a5a87e61e960" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 38093 } } peers { permanent_uuid: "06b8cf35c542444a891be6483830f742" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37317 } } peers { permanent_uuid: "50530183307349a9b135d15862167300" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 35729 } }
I20260925 22:41:49.966895 13058 ts_tablet_manager.cc:1428] T 5614bacc4579446793ad8349372c58b0 P 5df199e5ba78424cbbf3a5a87e61e960: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:41:49.967470 13059 ts_tablet_manager.cc:1428] T 5614bacc4579446793ad8349372c58b0 P 50530183307349a9b135d15862167300: Time spent starting tablet: real 0.007s user 0.002s sys 0.006s
I20260925 22:41:49.967535 12773 heartbeater.cc:503] Master 127.2.74.254:43001 was elected leader, sending a full tablet report...
I20260925 22:41:49.970918 13061 raft_consensus.cc:348] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5df199e5ba78424cbbf3a5a87e61e960" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 38093 } } peers { permanent_uuid: "06b8cf35c542444a891be6483830f742" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37317 } } peers { permanent_uuid: "50530183307349a9b135d15862167300" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 35729 } }
I20260925 22:41:49.971228 13061 raft_consensus.cc:374] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:41:49.971315 13061 raft_consensus.cc:729] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06b8cf35c542444a891be6483830f742, State: Initialized, Role: FOLLOWER
I20260925 22:41:49.971823 13061 consensus_queue.cc:260] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5df199e5ba78424cbbf3a5a87e61e960" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 38093 } } peers { permanent_uuid: "06b8cf35c542444a891be6483830f742" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37317 } } peers { permanent_uuid: "50530183307349a9b135d15862167300" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 35729 } }
I20260925 22:41:49.972543 12907 heartbeater.cc:503] Master 127.2.74.254:43001 was elected leader, sending a full tablet report...
I20260925 22:41:49.972841 13061 ts_tablet_manager.cc:1428] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260925 22:41:50.130613 13066 raft_consensus.cc:482] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:41:50.130810 13066 raft_consensus.cc:504] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5df199e5ba78424cbbf3a5a87e61e960" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 38093 } } peers { permanent_uuid: "06b8cf35c542444a891be6483830f742" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37317 } } peers { permanent_uuid: "50530183307349a9b135d15862167300" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 35729 } }
I20260925 22:41:50.132047 13066 leader_election.cc:288] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5df199e5ba78424cbbf3a5a87e61e960 (127.2.74.195:38093), 50530183307349a9b135d15862167300 (127.2.74.193:35729)
W20260925 22:41:50.132246 13042 tablet.cc:2369] T 5614bacc4579446793ad8349372c58b0 P 5df199e5ba78424cbbf3a5a87e61e960: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:41:50.137470 13053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:50.132572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:55163 (local address 127.2.74.195:38093)
0925 22:41:50.132716 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:50.132721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:50.133099 (+ 378us) server_negotiation.cc:407] Connection header received
0925 22:41:50.133206 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:50.133208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:50.133248 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:50.133305 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:50.133567 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:50.134283 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:50.136055 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:50.136677 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:50.136808 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:50.136864 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:50.137233 (+ 369us) server_negotiation.cc:299] Negotiation successful
0925 22:41:50.137324 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 22:41:50.137470 13056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:50.132730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:43087 (local address 127.2.74.193:35729)
0925 22:41:50.132855 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:50.132858 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:50.132871 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:41:50.132918 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:50.132921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:50.132959 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:50.133025 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:41:50.133411 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:50.134212 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:50.136055 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:50.136677 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:50.136808 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:50.136865 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:50.137233 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 22:41:50.137324 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 22:41:50.137475 12909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:50.132329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:35729 (local address 127.2.74.194:43087)
0925 22:41:50.132535 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:50.132552 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:50.132659 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:50.133093 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:50.133096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:50.133108 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:50.133248 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:50.133254 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:50.134369 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:50.134374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:50.135893 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:50.135905 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:50.136847 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:50.136851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:50.137049 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:50.137079 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 22:41:50.137164 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 22:41:50.137267 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":226}
I20260925 22:41:50.137482 13067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:50.132326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:38093 (local address 127.2.74.194:55163)
0925 22:41:50.132947 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:50.132964 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:41:50.133106 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:41:50.133346 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:41:50.133352 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:41:50.133371 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:41:50.133473 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:50.133477 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:50.134395 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:50.134397 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:50.135893 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:41:50.135905 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:41:50.136847 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:41:50.136851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:41:50.137034 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:50.137083 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 22:41:50.137164 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 22:41:50.137267 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":230,"thread_start_us":105,"threads_started":1}
I20260925 22:41:50.138228 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5614bacc4579446793ad8349372c58b0" candidate_uuid: "06b8cf35c542444a891be6483830f742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50530183307349a9b135d15862167300" is_pre_election: true
I20260925 22:41:50.138386 12995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5614bacc4579446793ad8349372c58b0" candidate_uuid: "06b8cf35c542444a891be6483830f742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5df199e5ba78424cbbf3a5a87e61e960" is_pre_election: true
I20260925 22:41:50.138576 12727 raft_consensus.cc:2454] T 5614bacc4579446793ad8349372c58b0 P 50530183307349a9b135d15862167300 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06b8cf35c542444a891be6483830f742 in term 0.
I20260925 22:41:50.138782 12995 raft_consensus.cc:2454] T 5614bacc4579446793ad8349372c58b0 P 5df199e5ba78424cbbf3a5a87e61e960 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06b8cf35c542444a891be6483830f742 in term 0.
I20260925 22:41:50.139159 12795 leader_election.cc:302] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06b8cf35c542444a891be6483830f742, 50530183307349a9b135d15862167300; no voters:
I20260925 22:41:50.139485 13066 raft_consensus.cc:2781] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:41:50.139592 13066 raft_consensus.cc:482] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:41:50.139658 13066 raft_consensus.cc:3037] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:50.140780 13066 raft_consensus.cc:504] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5df199e5ba78424cbbf3a5a87e61e960" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 38093 } } peers { permanent_uuid: "06b8cf35c542444a891be6483830f742" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37317 } } peers { permanent_uuid: "50530183307349a9b135d15862167300" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 35729 } }
I20260925 22:41:50.141177 13066 leader_election.cc:288] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742 [CANDIDATE]: Term 1 election: Requested vote from peers 5df199e5ba78424cbbf3a5a87e61e960 (127.2.74.195:38093), 50530183307349a9b135d15862167300 (127.2.74.193:35729)
I20260925 22:41:50.141628 12995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5614bacc4579446793ad8349372c58b0" candidate_uuid: "06b8cf35c542444a891be6483830f742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5df199e5ba78424cbbf3a5a87e61e960"
I20260925 22:41:50.141717 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5614bacc4579446793ad8349372c58b0" candidate_uuid: "06b8cf35c542444a891be6483830f742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50530183307349a9b135d15862167300"
I20260925 22:41:50.141777 12995 raft_consensus.cc:3037] T 5614bacc4579446793ad8349372c58b0 P 5df199e5ba78424cbbf3a5a87e61e960 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:50.141865 12727 raft_consensus.cc:3037] T 5614bacc4579446793ad8349372c58b0 P 50530183307349a9b135d15862167300 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:41:50.143178 12995 raft_consensus.cc:2454] T 5614bacc4579446793ad8349372c58b0 P 5df199e5ba78424cbbf3a5a87e61e960 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06b8cf35c542444a891be6483830f742 in term 1.
I20260925 22:41:50.143178 12727 raft_consensus.cc:2454] T 5614bacc4579446793ad8349372c58b0 P 50530183307349a9b135d15862167300 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06b8cf35c542444a891be6483830f742 in term 1.
I20260925 22:41:50.143494 12794 leader_election.cc:302] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06b8cf35c542444a891be6483830f742, 5df199e5ba78424cbbf3a5a87e61e960; no voters:
I20260925 22:41:50.143702 13066 raft_consensus.cc:2781] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:41:50.143941 13066 raft_consensus.cc:686] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742 [term 1 LEADER]: Becoming Leader. State: Replica: 06b8cf35c542444a891be6483830f742, State: Running, Role: LEADER
I20260925 22:41:50.144356 13066 consensus_queue.cc:237] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5df199e5ba78424cbbf3a5a87e61e960" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 38093 } } peers { permanent_uuid: "06b8cf35c542444a891be6483830f742" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37317 } } peers { permanent_uuid: "50530183307349a9b135d15862167300" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 35729 } }
I20260925 22:41:50.147697 12582 catalog_manager.cc:5495] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06b8cf35c542444a891be6483830f742 (127.2.74.194). New cstate: current_term: 1 leader_uuid: "06b8cf35c542444a891be6483830f742" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5df199e5ba78424cbbf3a5a87e61e960" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 38093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06b8cf35c542444a891be6483830f742" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50530183307349a9b135d15862167300" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 35729 } health_report { overall_health: UNKNOWN } } }
W20260925 22:41:50.154458 12908 tablet.cc:2369] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:41:50.163409 12774 tablet.cc:2369] T 5614bacc4579446793ad8349372c58b0 P 50530183307349a9b135d15862167300: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:41:50.182281 13057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:50.178767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38600 (local address 127.2.74.194:37317)
0925 22:41:50.178885 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 22:41:50.178888 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:41:50.178902 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:41:50.179053 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:41:50.179056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:41:50.179112 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:41:50.179205 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:41:50.179395 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:50.179910 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:50.180803 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:41:50.181018 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:41:50.181097 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:41:50.181363 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:41:50.181837 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:41:50.181931 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 22:41:50.182007 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 22:41:50.182061 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 22:41:50.188618 12727 raft_consensus.cc:1261] T 5614bacc4579446793ad8349372c58b0 P 50530183307349a9b135d15862167300 [term 1 FOLLOWER]: Refusing update from remote peer 06b8cf35c542444a891be6483830f742: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:41:50.188627 12995 raft_consensus.cc:1261] T 5614bacc4579446793ad8349372c58b0 P 5df199e5ba78424cbbf3a5a87e61e960 [term 1 FOLLOWER]: Refusing update from remote peer 06b8cf35c542444a891be6483830f742: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:41:50.189461 13066 consensus_queue.cc:1035] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50530183307349a9b135d15862167300" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 35729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:41:50.189666 13068 consensus_queue.cc:1035] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5df199e5ba78424cbbf3a5a87e61e960" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 38093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:41:50.198097 13073 mvcc.cc:205] Tried to move back new op lower bound from 7333380547323805696 to 7333380547154309120. Current Snapshot: MvccSnapshot[applied={T|T < 7333380547323805696}]
I20260925 22:41:50.206730 12580 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:41:50.206988 12580 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:41:50.209661 12580 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f32c24fc95d543a1a39676f9b4fc5171: Sending DeleteTablet for 3 replicas of tablet 5614bacc4579446793ad8349372c58b0
I20260925 22:41:50.210587 12841 tablet_service.cc:1514] Processing DeleteTablet for tablet 5614bacc4579446793ad8349372c58b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:50 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:38588
I20260925 22:41:50.210602 12707 tablet_service.cc:1514] Processing DeleteTablet for tablet 5614bacc4579446793ad8349372c58b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:50 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:44950
I20260925 22:41:50.211189 12975 tablet_service.cc:1514] Processing DeleteTablet for tablet 5614bacc4579446793ad8349372c58b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:41:50 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:58246
I20260925 22:41:50.211391 13082 tablet_replica.cc:330] T 5614bacc4579446793ad8349372c58b0 P 50530183307349a9b135d15862167300: stopping tablet replica
I20260925 22:41:50.211517 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 12644
I20260925 22:41:50.212773 13083 tablet_replica.cc:330] T 5614bacc4579446793ad8349372c58b0 P 5df199e5ba78424cbbf3a5a87e61e960: stopping tablet replica
I20260925 22:41:50.213106 13083 raft_consensus.cc:2229] T 5614bacc4579446793ad8349372c58b0 P 5df199e5ba78424cbbf3a5a87e61e960 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:41:50.213454 13083 raft_consensus.cc:2258] T 5614bacc4579446793ad8349372c58b0 P 5df199e5ba78424cbbf3a5a87e61e960 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:41:50.213501 13081 tablet_replica.cc:330] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742: stopping tablet replica
I20260925 22:41:50.213917 13081 raft_consensus.cc:2229] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:41:50.214401 13081 raft_consensus.cc:2258] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:41:50.215255 13083 ts_tablet_manager.cc:1905] T 5614bacc4579446793ad8349372c58b0 P 5df199e5ba78424cbbf3a5a87e61e960: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:41:50.216208 13081 ts_tablet_manager.cc:1905] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:41:50.218277 13083 ts_tablet_manager.cc:1918] T 5614bacc4579446793ad8349372c58b0 P 5df199e5ba78424cbbf3a5a87e61e960: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:41:50.218391 13083 log.cc:1196] T 5614bacc4579446793ad8349372c58b0 P 5df199e5ba78424cbbf3a5a87e61e960: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-2/wal/wals/5614bacc4579446793ad8349372c58b0
I20260925 22:41:50.218724 13083 ts_tablet_manager.cc:1939] T 5614bacc4579446793ad8349372c58b0 P 5df199e5ba78424cbbf3a5a87e61e960: Deleting consensus metadata
I20260925 22:41:50.220026 13081 ts_tablet_manager.cc:1918] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:41:50.220168 13081 log.cc:1196] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376047580784-2347-0/minicluster-data/ts-1/wal/wals/5614bacc4579446793ad8349372c58b0
I20260925 22:41:50.220167 12566 catalog_manager.cc:4841] TS 5df199e5ba78424cbbf3a5a87e61e960 (127.2.74.195:38093): tablet 5614bacc4579446793ad8349372c58b0 (table test-table [id=e756eaaca93745dbaf8db64cbd6484ef]) successfully deleted
I20260925 22:41:50.220516 13081 ts_tablet_manager.cc:1939] T 5614bacc4579446793ad8349372c58b0 P 06b8cf35c542444a891be6483830f742: Deleting consensus metadata
W20260925 22:41:50.220516 12567 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.74.193:35729 (error 108)
I20260925 22:41:50.221390 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 12778
I20260925 22:41:50.221643 13055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:50.221302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:35729 (local address 127.0.0.1:44952)
0925 22:41:50.221454 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:50.221537 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.193:35729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 22:41:50.222612 12567 catalog_manager.cc:4584] TS 50530183307349a9b135d15862167300 (127.2.74.193:35729): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5614bacc4579446793ad8349372c58b0: Network error: Client connection negotiation failed: client connection to 127.2.74.193:35729: connect: Connection refused (error 111)
I20260925 22:41:50.227567 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 12912
I20260925 22:41:50.228593 13055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:41:50.227764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:37317 (local address 127.0.0.1:38604)
0925 22:41:50.228397 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0925 22:41:50.228474 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.194:37317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505}
I20260925 22:41:50.233278 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 12550
2026-09-25T22:41:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2529 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:41:50 dist-test-slave-csnl krb5kdc[13087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:41:50 dist-test-slave-csnl krb5kdc[13087](info): set up 2 sockets
Sep 25 22:41:50 dist-test-slave-csnl krb5kdc[13087](info): commencing operation
krb5kdc: starting...
W20260925 22:41:51.292639 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:41:51 dist-test-slave-csnl krb5kdc[13087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:41:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:41:51Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.74.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.74.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.74.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/rangeradmin_127.2.74.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.74.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/rangeradmin_127.2.74.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.74.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.74.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.74.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/rangerlookup_127.2.74.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.74.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/rangerlookup_127.2.74.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.74.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.74.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.74.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/HTTP_127.2.74.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.74.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/HTTP_127.2.74.212@KRBTEST.COM.keytab.
I20260925 22:41:51.468241 2347 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskx4wBtU/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/postgres -l logfile start
2026-09-25 22:41:54.184 UTC [13117] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 22:41:54.185 UTC [13117] LOG: listening on IPv4 address "127.2.74.212", port 45115
2026-09-25 22:41:54.188 UTC [13117] LOG: listening on Unix socket "/tmp/.s.PGSQL.45115"
2026-09-25 22:41:54.209 UTC [13120] LOG: database system was shut down at 2026-09-25 22:41:52 UTC
2026-09-25 22:41:54.213 UTC [13117] LOG: database system is ready to accept connections
I20260925 22:41:55.164014 2347 mini_postgres.cc:96] Postgres bound to 45115
2026-09-25 22:41:55.170 UTC [13128] FATAL: database "slave" does not exist
127.2.74.212:45115 - accepting connections
I20260925 22:41:55.171682 2347 mini_ranger.cc:161] Starting Ranger...
I20260925 22:41:55.187757 2347 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 22:41:55.712602 2347 mini_ranger.cc:87] Created ranger Postgres database
I20260925 22:41:55.712713 2347 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ranger-admin
2026-09-25 22:41:56,002 [I] DB FLAVOR :POSTGRES
2026-09-25 22:41:56,002 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 22:41:56,002 [I] Checking connection..
2026-09-25 22:41:56,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 22:41:56,271 [I] Checking connection passed.
2026-09-25 22:41:56,272 [I] --------- Verifying version history table ---------
2026-09-25 22:41:56,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:41:56,527 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 22:41:56,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:41:56,795 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 22:41:56,795 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 22:41:56,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 22:41:57.064 UTC [13192] WARNING: there is no transaction in progress
2026-09-25 22:41:57,071 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 22:41:57,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:41:57,327 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 22:41:57,327 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 22:41:57,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 22:41:57,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 22:41:57,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-csnl.c.gcp-upstream.internal','N') ;"
2026-09-25 22:41:58,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 22:41:58,350 [I] Table x_portal_user does not exist in database ranger
2026-09-25 22:41:58,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 22:41:58,601 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 22:41:58,601 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 22:41:58,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 22:41:59,128 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 22:41:59,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 22:42:00.164 UTC [13300] WARNING: there is no transaction in progress
2026-09-25 22:42:00.181 UTC [13300] WARNING: there is no transaction in progress
2026-09-25 22:42:00.198 UTC [13300] WARNING: there is no transaction in progress
2026-09-25 22:42:00.214 UTC [13300] WARNING: there is no transaction in progress
2026-09-25 22:42:00.231 UTC [13300] WARNING: there is no transaction in progress
2026-09-25 22:42:00.249 UTC [13300] WARNING: there is no transaction in progress
2026-09-25 22:42:00.309 UTC [13300] WARNING: there is no transaction in progress
2026-09-25 22:42:00.318 UTC [13300] WARNING: there is no transaction in progress
2026-09-25 22:42:00.335 UTC [13300] WARNING: there is no transaction in progress
2026-09-25 22:42:00.346 UTC [13300] WARNING: there is no transaction in progress
2026-09-25 22:42:00.358 UTC [13300] WARNING: there is no transaction in progress
2026-09-25 22:42:00.373 UTC [13300] WARNING: there is no transaction in progress
2026-09-25 22:42:00.385 UTC [13300] WARNING: there is no transaction in progress
2026-09-25 22:42:00.393 UTC [13300] WARNING: there is no transaction in progress
2026-09-25 22:42:00.401 UTC [13300] WARNING: there is no transaction in progress
2026-09-25 22:42:00.409 UTC [13300] WARNING: there is no transaction in progress
2026-09-25 22:42:01,140 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 22:42:01,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 22:42:01,398 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 22:42:01,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 22:42:01,642 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 22:42:01,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 22:42:01,896 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 22:42:01,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 22:42:02,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 22:42:02,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-csnl.c.gcp-upstream.internal';"
2026-09-25 22:42:02,658 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 22:42:02,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 22:42:02,910 [I] DB_PATCHES have already been applied
I20260925 22:42:02.916302 2347 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ranger-admin:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskx4wBtU/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 22:42:02.916378 2347 mini_ranger.cc:193] Using host: 127.2.74.212
I20260925 22:42:02.919900 2347 mini_ranger.cc:227] Ranger admin URL: http://127.2.74.212:34135
Sep 25, 2026 10:42:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:42:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:42:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 10:42:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 10:42:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 10:42:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 10:42:03 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 22:42:04.051612 2347 mini_ranger.cc:160] Time spent starting Ranger: real 8.880s user 0.005s sys 0.005s
Sep 25, 2026 10:42:24 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 10:42:24 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 10:42:24 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 10:42:24 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 10:42:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 10:42:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 10:42:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 10:42:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 10:42:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 10:42:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 10:42:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 10:42:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 10:42:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 10:42:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 10:42:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 10:42:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 10:42:24 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 10:42:25 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 22:42:26.201226 2347 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.74.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.74.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.74.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/rangerkms_127.2.74.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.74.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/rangerkms_127.2.74.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.74.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.74.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.74.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/HTTP_127.2.74.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.74.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/HTTP_127.2.74.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 22:42:26 dist-test-slave-csnl krb5kdc[13087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376146, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 22:42:26.290796 2347 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 22:42:26.301517 2347 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 22:42:27.256846 2347 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 22:42:27.256937 2347 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ranger-kms
I20260925 22:42:27.257043 2347 mini_ranger_kms.cc:227] Using postgres at 127.2.74.212:45115
2026-09-25 22:42:27,463 [I] DB FLAVOR :POSTGRES
2026-09-25 22:42:27,463 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 22:42:27,464 [I] Checking connection
2026-09-25 22:42:27,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:45115/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 22:42:27,749 [I] connection success
2026-09-25 22:42:27,750 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 22:42:27,750 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 22:42:27,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:45115/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 22:42:27,995 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 22:42:27,996 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 22:42:27,996 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 22:42:27,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:45115/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 22:42:28,268 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 22:42:28.796904 2347 mini_ranger_kms.cc:310] Created kms service
I20260925 22:42:29.009819 2347 mini_ranger_kms.cc:326] Created kudu user
I20260925 22:42:29.065510 2347 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 22:42:29 dist-test-slave-csnl krb5kdc[13087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376146, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.74.212@KRBTEST.COM
I20260925 22:42:29.504556 2347 mini_ranger_kms.cc:384] Added ranger policy
I20260925 22:42:29.504733 2347 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ranger-kms:/tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 22:42:29.504762 2347 mini_ranger_kms.cc:240] Using host: 127.2.74.212
I20260925 22:42:29.508462 2347 mini_ranger_kms.cc:276] Ranger KMS PID: 13480
I20260925 22:42:29.508631 2347 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.74.212:48059
Sep 25, 2026 10:42:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:42:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:42:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 10:42:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 10:42:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 10:42:31 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 22:42:31.783888 2347 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.493s user 0.023s sys 0.166s
I20260925 22:42:31.784029 2347 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 22:42:31.784143 2347 mini_ranger_kms.cc:401] 127.2.74.212:48059/kms/v1/keys
Sep 25 22:42:31 dist-test-slave-csnl krb5kdc[13087](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790376151, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.74.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 22:42:32 dist-test-slave-csnl krb5kdc[13087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376151, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.74.212@KRBTEST.COM for HTTP/127.2.74.212@KRBTEST.COM
Sep 25 22:42:32 dist-test-slave-csnl krb5kdc[13087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376151, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.74.212@KRBTEST.COM for HTTP/127.2.74.212@KRBTEST.COM
Sep 25 22:42:35 dist-test-slave-csnl krb5kdc[13087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:42:35.063894 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:35039
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:35471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:35039
--ranger_config_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.74.212:48059/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:42:35.188679 13531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:42:35.188998 13531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:42:35.189040 13531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:42:35.193451 13531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:42:35.193542 13531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:42:35.193562 13531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:42:35.193580 13531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:42:35.193596 13531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:42:35.199681 13531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:35471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.74.212:48059/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:35039
--ranger_config_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:35039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:42:35.200887 13531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:42:35.202028 13531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:42:35.211875 13531 server_base.cc:1034] running on GCE node
W20260925 22:42:35.211814 13536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:42:35.211814 13539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:42:35.211818 13537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:42:35.212832 13531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:42:35.214104 13531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:42:35.215309 13531 hybrid_clock.cc:648] HybridClock initialized: now 1790376155215291 us; error 35 us; skew 500 ppm
Sep 25 22:42:35 dist-test-slave-csnl krb5kdc[13087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376155, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:42:35.222244 13531 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:42:35.223471 13531 webserver.cc:460] Webserver started at http://127.2.74.254:34849/ using document root <none> and password file <none>
I20260925 22:42:35.224105 13531 fs_manager.cc:359] Metadata directory not provided
I20260925 22:42:35.224201 13531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:42:35.224440 13531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:42:35 dist-test-slave-csnl krb5kdc[13087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376155, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for HTTP/127.2.74.212@KRBTEST.COM
I20260925 22:42:35.285096 13531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "8bc436dd21774e54b3efc0e181377bb2"
format_stamp: "Formatted at 2026-09-25 22:42:35 on dist-test-slave-csnl"
server_key: "5312a93c4823ac8104d216a0992dc64d"
server_key_iv: "4d571440fadc4539f6b07903b9513269"
server_key_version: "kuduclusterkey@0"
I20260925 22:42:35.285794 13531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "8bc436dd21774e54b3efc0e181377bb2"
format_stamp: "Formatted at 2026-09-25 22:42:35 on dist-test-slave-csnl"
server_key: "5312a93c4823ac8104d216a0992dc64d"
server_key_iv: "4d571440fadc4539f6b07903b9513269"
server_key_version: "kuduclusterkey@0"
I20260925 22:42:35.290506 13531 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 22:42:35.311950 13546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:42:35.313982 13531 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:42:35.314196 13531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "8bc436dd21774e54b3efc0e181377bb2"
format_stamp: "Formatted at 2026-09-25 22:42:35 on dist-test-slave-csnl"
server_key: "5312a93c4823ac8104d216a0992dc64d"
server_key_iv: "4d571440fadc4539f6b07903b9513269"
server_key_version: "kuduclusterkey@0"
I20260925 22:42:35.314361 13531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:42:35.337456 13531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:42:35.340883 13531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:42:35.341185 13531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:42:35.350101 13531 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:35039
I20260925 22:42:35.350128 13598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:35039 every 8 connection(s)
I20260925 22:42:35.351279 13531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:42:35.354329 13599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:42:35.354446 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 13531
I20260925 22:42:35.354646 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:42:35.355046 2347 external_mini_cluster.cc:1428] Setting key 79388316620986ab2ef83c8ab307ec67
I20260925 22:42:35.361466 13599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bc436dd21774e54b3efc0e181377bb2: Bootstrap starting.
I20260925 22:42:35.364207 13599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bc436dd21774e54b3efc0e181377bb2: Neither blocks nor log segments found. Creating new log.
Sep 25 22:42:35 dist-test-slave-csnl krb5kdc[13087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:42:35.365159 13599 log.cc:824] T 00000000000000000000000000000000 P 8bc436dd21774e54b3efc0e181377bb2: Log is configured to *not* fsync() on all Append() calls
I20260925 22:42:35.368165 13599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bc436dd21774e54b3efc0e181377bb2: No bootstrap required, opened a new log
I20260925 22:42:35.371901 13602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:42:35.356687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50658 (local address 127.2.74.254:35039)
0925 22:42:35.357157 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0925 22:42:35.357172 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:42:35.357211 (+ 39us) server_negotiation.cc:407] Connection header received
0925 22:42:35.357948 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:42:35.357967 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:42:35.358330 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:42:35.358705 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:42:35.359613 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:42:35.360637 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:42:35.361595 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:42:35.362180 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:42:35.362389 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:42:35.365522 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:42:35.365530 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:42:35.365551 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:42:35.365595 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:42:35.368403 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:42:35.368937 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:42:35.368944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:42:35.368951 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:42:35.369097 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:42:35.369403 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:42:35.369407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:42:35.369409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:42:35.369957 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:42:35.370172 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 22:42:35.370513 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 22:42:35.370826 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":134,"threads_started":1}
I20260925 22:42:35.372417 13599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bc436dd21774e54b3efc0e181377bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc436dd21774e54b3efc0e181377bb2" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 35039 } }
I20260925 22:42:35.372696 13599 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bc436dd21774e54b3efc0e181377bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:42:35.372743 13599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bc436dd21774e54b3efc0e181377bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bc436dd21774e54b3efc0e181377bb2, State: Initialized, Role: FOLLOWER
I20260925 22:42:35.373231 13599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bc436dd21774e54b3efc0e181377bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc436dd21774e54b3efc0e181377bb2" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 35039 } }
I20260925 22:42:35.373409 13599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bc436dd21774e54b3efc0e181377bb2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:42:35.373564 13599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bc436dd21774e54b3efc0e181377bb2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:42:35.373694 13599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bc436dd21774e54b3efc0e181377bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:42:35.374897 13599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bc436dd21774e54b3efc0e181377bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc436dd21774e54b3efc0e181377bb2" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 35039 } }
I20260925 22:42:35.375325 13599 leader_election.cc:302] T 00000000000000000000000000000000 P 8bc436dd21774e54b3efc0e181377bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bc436dd21774e54b3efc0e181377bb2; no voters:
I20260925 22:42:35.375694 13599 leader_election.cc:288] T 00000000000000000000000000000000 P 8bc436dd21774e54b3efc0e181377bb2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:42:35.375851 13604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bc436dd21774e54b3efc0e181377bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:42:35.376214 13604 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bc436dd21774e54b3efc0e181377bb2 [term 1 LEADER]: Becoming Leader. State: Replica: 8bc436dd21774e54b3efc0e181377bb2, State: Running, Role: LEADER
I20260925 22:42:35.376678 13604 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bc436dd21774e54b3efc0e181377bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc436dd21774e54b3efc0e181377bb2" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 35039 } }
I20260925 22:42:35.377317 13599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bc436dd21774e54b3efc0e181377bb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:42:35.378593 13606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc436dd21774e54b3efc0e181377bb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bc436dd21774e54b3efc0e181377bb2. Latest consensus state: current_term: 1 leader_uuid: "8bc436dd21774e54b3efc0e181377bb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc436dd21774e54b3efc0e181377bb2" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 35039 } } }
I20260925 22:42:35.378765 13606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc436dd21774e54b3efc0e181377bb2 [sys.catalog]: This master's current role is: LEADER
I20260925 22:42:35.379209 13613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:42:35.379227 13605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc436dd21774e54b3efc0e181377bb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bc436dd21774e54b3efc0e181377bb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc436dd21774e54b3efc0e181377bb2" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 35039 } } }
I20260925 22:42:35.379402 13605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc436dd21774e54b3efc0e181377bb2 [sys.catalog]: This master's current role is: LEADER
I20260925 22:42:35.379742 13599 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 22:42:35.386271 13613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:42:35.395665 13613 catalog_manager.cc:1347] Generated new cluster ID: 1dbe61e096494f84bb9f633a8842e8ec
I20260925 22:42:35.395750 13613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:42:35.407526 13613 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:42:35.408942 13613 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:42:35.415792 13613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bc436dd21774e54b3efc0e181377bb2: Generated new TSK 0
I20260925 22:42:35.416848 13613 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
I20260925 22:42:36.139326 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.74.212:48059/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:0
--local_ip_for_outbound_sockets=127.2.74.193
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:35039
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:35471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:42:36.291354 13644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:42:36.291712 13644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:42:36.291791 13644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:42:36.296185 13644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:42:36.296293 13644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:42:36.296404 13644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:42:36.302541 13644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:35471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.74.212:48059/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:35039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:42:36.303828 13644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:42:36.304962 13644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:42:36.318120 13653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:42:36.318650 13651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:42:36.318785 13650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:42:36.318660 13644 server_base.cc:1034] running on GCE node
I20260925 22:42:36.319407 13644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:42:36.320295 13644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:42:36.321533 13644 hybrid_clock.cc:648] HybridClock initialized: now 1790376156321506 us; error 44 us; skew 500 ppm
Sep 25 22:42:36 dist-test-slave-csnl krb5kdc[13087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376156, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:42:36.325372 13644 init.cc:377] Logged in from keytab as kudu/127.2.74.193@KRBTEST.COM (short username kudu)
I20260925 22:42:36.326476 13644 webserver.cc:460] Webserver started at http://127.2.74.193:42565/ using document root <none> and password file <none>
I20260925 22:42:36.327208 13644 fs_manager.cc:359] Metadata directory not provided
I20260925 22:42:36.327313 13644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:42:36.327600 13644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:42:36 dist-test-slave-csnl krb5kdc[13087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376156, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for HTTP/127.2.74.212@KRBTEST.COM
I20260925 22:42:36.348368 13644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-0/data/instance:
uuid: "5bc4a646adb74c05ba923c9939e46643"
format_stamp: "Formatted at 2026-09-25 22:42:36 on dist-test-slave-csnl"
server_key: "f149dd0733b7c28329b2f2df42136472"
server_key_iv: "226365d35ecdda2697f3f0e0df0470da"
server_key_version: "kuduclusterkey@0"
I20260925 22:42:36.349043 13644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance:
uuid: "5bc4a646adb74c05ba923c9939e46643"
format_stamp: "Formatted at 2026-09-25 22:42:36 on dist-test-slave-csnl"
server_key: "f149dd0733b7c28329b2f2df42136472"
server_key_iv: "226365d35ecdda2697f3f0e0df0470da"
server_key_version: "kuduclusterkey@0"
I20260925 22:42:36.353933 13644 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 22:42:36.372669 13661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:42:36.374902 13644 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:42:36.375123 13644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "5bc4a646adb74c05ba923c9939e46643"
format_stamp: "Formatted at 2026-09-25 22:42:36 on dist-test-slave-csnl"
server_key: "f149dd0733b7c28329b2f2df42136472"
server_key_iv: "226365d35ecdda2697f3f0e0df0470da"
server_key_version: "kuduclusterkey@0"
I20260925 22:42:36.375299 13644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:42:36.401304 13644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:42:36.405524 13644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:42:36.405807 13644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:42:36.406594 13644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:42:36.407864 13644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:42:36.407980 13644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:42:36.408094 13644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:42:36.408150 13644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:42:36.420266 13644 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:40537
I20260925 22:42:36.421638 13644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:42:36.423704 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 13644
I20260925 22:42:36.423861 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance
I20260925 22:42:36.424211 2347 external_mini_cluster.cc:1428] Setting key db63f72d199de8a90398d8f568394e58
I20260925 22:42:36.425290 13774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:40537 every 8 connection(s)
WARNING: no policy specified for kudu/127.2.74.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.194@KRBTEST.COM" created.
Sep 25 22:42:36 dist-test-slave-csnl krb5kdc[13087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376156, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:42:36.450340 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:42:36.427775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:35039 (local address 127.2.74.193:37719)
0925 22:42:36.433460 (+ 5685us) negotiation.cc:107] Waiting for socket to connect
0925 22:42:36.433551 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0925 22:42:36.434885 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:42:36.436375 (+ 1490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:42:36.436385 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:42:36.436798 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:42:36.437408 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:42:36.437424 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:42:36.438467 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:42:36.438471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:42:36.439833 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:42:36.439851 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:42:36.440805 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:42:36.440811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:42:36.441084 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:42:36.441912 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:42:36.441930 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:42:36.443498 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:42:36.446364 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:42:36.446371 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:42:36.446390 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:42:36.446730 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:42:36.447166 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:42:36.447172 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:42:36.447176 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:42:36.447495 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:42:36.448109 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:42:36.448120 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:42:36.448555 (+ 435us) client_negotiation.cc:769] Sending connection context
0925 22:42:36.448786 (+ 231us) client_negotiation.cc:240] Negotiation successful
0925 22:42:36.449187 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5382,"thread_start_us":154,"threads_started":1}
I20260925 22:42:36.451334 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:42:36.427515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:37719 (local address 127.2.74.254:35039)
0925 22:42:36.433076 (+ 5561us) server_negotiation.cc:206] Beginning negotiation
0925 22:42:36.433084 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:42:36.433685 (+ 601us) server_negotiation.cc:407] Connection header received
0925 22:42:36.435241 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:42:36.435248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:42:36.435331 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:42:36.435410 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:42:36.437611 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:42:36.438299 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:42:36.440020 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:42:36.440444 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:42:36.440548 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:42:36.443745 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:42:36.443749 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:42:36.443751 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:42:36.443780 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:42:36.446160 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:42:36.446888 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:42:36.446894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:42:36.446897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:42:36.447039 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:42:36.447638 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:42:36.447642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:42:36.447644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:42:36.447940 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:42:36.448062 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 22:42:36.450852 (+ 2790us) server_negotiation.cc:299] Negotiation successful
0925 22:42:36.451156 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5412,"thread_start_us":109,"threads_started":1}
I20260925 22:42:36.453617 13775 heartbeater.cc:347] Connected to a master server at 127.2.74.254:35039
I20260925 22:42:36.453886 13775 heartbeater.cc:464] Registering TS with master...
I20260925 22:42:36.454520 13775 heartbeater.cc:511] Master 127.2.74.254:35039 requested a full tablet report, sending...
I20260925 22:42:36.456593 13563 ts_manager.cc:194] Registered new tserver with Master: 5bc4a646adb74c05ba923c9939e46643 (127.2.74.193:40537)
I20260925 22:42:36.458220 13563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.193@KRBTEST.COM'} at 127.2.74.193:37719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
I20260925 22:42:36.470909 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.74.212:48059/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:0
--local_ip_for_outbound_sockets=127.2.74.194
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:35039
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:35471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:42:36 dist-test-slave-csnl krb5kdc[13087](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790376156, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 22:42:36.605809 13783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:42:36.606175 13783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:42:36.606287 13783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:42:36.611243 13783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:42:36.611343 13783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:42:36.611445 13783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:42:36.617523 13783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:35471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.74.212:48059/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:35039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:42:36.618943 13783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:42:36.620251 13783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:42:36.632021 13792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:42:36.632041 13790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:42:36.632046 13789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:42:36.632609 13783 server_base.cc:1034] running on GCE node
I20260925 22:42:36.633296 13783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:42:36.634119 13783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:42:36.635299 13783 hybrid_clock.cc:648] HybridClock initialized: now 1790376156635262 us; error 61 us; skew 500 ppm
Sep 25 22:42:36 dist-test-slave-csnl krb5kdc[13087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376156, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:42:36.638510 13783 init.cc:377] Logged in from keytab as kudu/127.2.74.194@KRBTEST.COM (short username kudu)
I20260925 22:42:36.639801 13783 webserver.cc:460] Webserver started at http://127.2.74.194:41749/ using document root <none> and password file <none>
I20260925 22:42:36.640565 13783 fs_manager.cc:359] Metadata directory not provided
I20260925 22:42:36.640645 13783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:42:36.640877 13783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:42:36 dist-test-slave-csnl krb5kdc[13087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376156, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for HTTP/127.2.74.212@KRBTEST.COM
I20260925 22:42:36.657830 13783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-1/data/instance:
uuid: "9f6fce6b8c8b4506b17475cd495021b0"
format_stamp: "Formatted at 2026-09-25 22:42:36 on dist-test-slave-csnl"
server_key: "e8dc61d0e90d5327bbd41f29d117ff03"
server_key_iv: "c22d08b0cad4e15a0c89b6937f93ba00"
server_key_version: "kuduclusterkey@0"
I20260925 22:42:36.658579 13783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance:
uuid: "9f6fce6b8c8b4506b17475cd495021b0"
format_stamp: "Formatted at 2026-09-25 22:42:36 on dist-test-slave-csnl"
server_key: "e8dc61d0e90d5327bbd41f29d117ff03"
server_key_iv: "c22d08b0cad4e15a0c89b6937f93ba00"
server_key_version: "kuduclusterkey@0"
I20260925 22:42:36.663002 13783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260925 22:42:36.679759 13799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:42:36.681440 13783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:42:36.681617 13783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "9f6fce6b8c8b4506b17475cd495021b0"
format_stamp: "Formatted at 2026-09-25 22:42:36 on dist-test-slave-csnl"
server_key: "e8dc61d0e90d5327bbd41f29d117ff03"
server_key_iv: "c22d08b0cad4e15a0c89b6937f93ba00"
server_key_version: "kuduclusterkey@0"
I20260925 22:42:36.681769 13783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:42:36.701098 13783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:42:36.705657 13783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:42:36.705888 13783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:42:36.706683 13783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:42:36.707816 13783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:42:36.707881 13783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:42:36.707935 13783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:42:36.707962 13783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:42:36.733194 13783 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:33377
I20260925 22:42:36.734483 13783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:42:36.737983 13912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:33377 every 8 connection(s)
I20260925 22:42:36.740069 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 13783
I20260925 22:42:36.740206 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance
I20260925 22:42:36.740547 2347 external_mini_cluster.cc:1428] Setting key c2f64bfac327790d91fe3503fb3dd529
Sep 25 22:42:36 dist-test-slave-csnl krb5kdc[13087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376156, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:42:36.760011 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:42:36.743992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:44649 (local address 127.2.74.254:35039)
0925 22:42:36.744133 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:42:36.744137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:42:36.744735 (+ 598us) server_negotiation.cc:407] Connection header received
0925 22:42:36.746016 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:42:36.746020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:42:36.746075 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:42:36.746129 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:42:36.747883 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:42:36.748752 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:42:36.751731 (+ 2979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:42:36.752123 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:42:36.752213 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:42:36.755650 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:42:36.755654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:42:36.755656 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:42:36.755685 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:42:36.757476 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:42:36.758147 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:42:36.758151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:42:36.758153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:42:36.758253 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:42:36.758664 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:42:36.758667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:42:36.758669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:42:36.758890 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:42:36.758967 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 22:42:36.759682 (+ 715us) server_negotiation.cc:299] Negotiation successful
0925 22:42:36.759825 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:42:36.762019 13915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:42:36.742738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:35039 (local address 127.2.74.194:44649)
0925 22:42:36.744568 (+ 1830us) negotiation.cc:107] Waiting for socket to connect
0925 22:42:36.744628 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 22:42:36.745737 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:42:36.746559 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:42:36.746569 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:42:36.746993 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:42:36.747692 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:42:36.747712 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:42:36.748923 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:42:36.748927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:42:36.750446 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:42:36.750457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:42:36.752454 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:42:36.752458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:42:36.752694 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:42:36.753570 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:42:36.753591 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:42:36.755432 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:42:36.757626 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:42:36.757633 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:42:36.757653 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:42:36.758022 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:42:36.758354 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:42:36.758357 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:42:36.758359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:42:36.758558 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:42:36.759065 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:42:36.759072 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:42:36.759396 (+ 324us) client_negotiation.cc:769] Sending connection context
0925 22:42:36.760482 (+ 1086us) client_negotiation.cc:240] Negotiation successful
0925 22:42:36.760869 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1599,"thread_start_us":198,"threads_started":1}
I20260925 22:42:36.763674 13913 heartbeater.cc:347] Connected to a master server at 127.2.74.254:35039
I20260925 22:42:36.763954 13913 heartbeater.cc:464] Registering TS with master...
I20260925 22:42:36.764526 13913 heartbeater.cc:511] Master 127.2.74.254:35039 requested a full tablet report, sending...
I20260925 22:42:36.765744 13563 ts_manager.cc:194] Registered new tserver with Master: 9f6fce6b8c8b4506b17475cd495021b0 (127.2.74.194:33377)
I20260925 22:42:36.766501 13563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.194@KRBTEST.COM'} at 127.2.74.194:44649
WARNING: no policy specified for kudu/127.2.74.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
I20260925 22:42:36.806449 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.74.212:48059/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:0
--local_ip_for_outbound_sockets=127.2.74.195
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:35039
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:35471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:42:36.978628 13918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:42:36.979173 13918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:42:36.979308 13918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:42:36.985868 13918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:42:36.986047 13918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:42:36.986186 13918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:42:36.994858 13918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:35471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.74.212:48059/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:35039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:42:36.996407 13918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:42:36.997901 13918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:42:37.011458 13918 server_base.cc:1034] running on GCE node
W20260925 22:42:37.011380 13924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:42:37.011842 13926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:42:37.012661 13923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:42:37.013276 13918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:42:37.014158 13918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:42:37.015618 13918 hybrid_clock.cc:648] HybridClock initialized: now 1790376157015543 us; error 89 us; skew 500 ppm
Sep 25 22:42:37 dist-test-slave-csnl krb5kdc[13087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376157, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:42:37.019517 13918 init.cc:377] Logged in from keytab as kudu/127.2.74.195@KRBTEST.COM (short username kudu)
I20260925 22:42:37.020843 13918 webserver.cc:460] Webserver started at http://127.2.74.195:45369/ using document root <none> and password file <none>
I20260925 22:42:37.021682 13918 fs_manager.cc:359] Metadata directory not provided
I20260925 22:42:37.021793 13918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:42:37.022068 13918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:42:37 dist-test-slave-csnl krb5kdc[13087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376157, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for HTTP/127.2.74.212@KRBTEST.COM
I20260925 22:42:37.048355 13918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-2/data/instance:
uuid: "6db2367a328d4ba6b59ec84f5bdfa3f6"
format_stamp: "Formatted at 2026-09-25 22:42:37 on dist-test-slave-csnl"
server_key: "e041236e3e8dbf1e1f208a142b7a8d51"
server_key_iv: "38a767fa9b6b9d264f218352daf8e70a"
server_key_version: "kuduclusterkey@0"
I20260925 22:42:37.049355 13918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance:
uuid: "6db2367a328d4ba6b59ec84f5bdfa3f6"
format_stamp: "Formatted at 2026-09-25 22:42:37 on dist-test-slave-csnl"
server_key: "e041236e3e8dbf1e1f208a142b7a8d51"
server_key_iv: "38a767fa9b6b9d264f218352daf8e70a"
server_key_version: "kuduclusterkey@0"
I20260925 22:42:37.061717 13918 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.004s sys 0.000s
Sep 25 22:42:37 dist-test-slave-csnl krb5kdc[13087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376156, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for HTTP/127.2.74.212@KRBTEST.COM
I20260925 22:42:37.089524 13933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:42:37.091521 13918 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.003s sys 0.000s
I20260925 22:42:37.091806 13918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "6db2367a328d4ba6b59ec84f5bdfa3f6"
format_stamp: "Formatted at 2026-09-25 22:42:37 on dist-test-slave-csnl"
server_key: "e041236e3e8dbf1e1f208a142b7a8d51"
server_key_iv: "38a767fa9b6b9d264f218352daf8e70a"
server_key_version: "kuduclusterkey@0"
I20260925 22:42:37.091989 13918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:42:37.125092 13918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:42:37.129190 13918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:42:37.129475 13918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:42:37.130790 13918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:42:37.132150 13918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:42:37.132279 13918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:42:37.132381 13918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:42:37.132443 13918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:42:37.144008 13918 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:36585
I20260925 22:42:37.145318 13918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:42:37.146787 14046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:36585 every 8 connection(s)
I20260925 22:42:37.155085 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 13918
I20260925 22:42:37.155216 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance
I20260925 22:42:37.155557 2347 external_mini_cluster.cc:1428] Setting key ca6b094414a79534350aa03e0150a77b
Sep 25 22:42:37 dist-test-slave-csnl krb5kdc[13087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376157, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:42:37.160665 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:42:37.147597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:53555 (local address 127.2.74.254:35039)
0925 22:42:37.147757 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 22:42:37.147762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:42:37.148783 (+ 1021us) server_negotiation.cc:407] Connection header received
0925 22:42:37.149840 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:42:37.149843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:42:37.149904 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:42:37.150020 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:42:37.151286 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:42:37.151833 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:42:37.153134 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:42:37.153373 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:42:37.153495 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:42:37.156373 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:42:37.156377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:42:37.156380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:42:37.156411 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:42:37.158291 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:42:37.158924 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:42:37.158927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:42:37.158929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:42:37.159059 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:42:37.159466 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:42:37.159470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:42:37.159472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:42:37.159683 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:42:37.159799 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 22:42:37.160349 (+ 550us) server_negotiation.cc:299] Negotiation successful
0925 22:42:37.160495 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 22:42:37.161672 14049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:42:37.148062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:35039 (local address 127.2.74.195:53555)
0925 22:42:37.148601 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0925 22:42:37.148664 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 22:42:37.149587 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:42:37.150213 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:42:37.150221 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:42:37.150519 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:42:37.151123 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:42:37.151140 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:42:37.151978 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:42:37.151982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:42:37.152993 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:42:37.153004 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:42:37.153556 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:42:37.153561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:42:37.153741 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:42:37.154407 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:42:37.154426 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:42:37.156167 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:42:37.158430 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:42:37.158438 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:42:37.158459 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:42:37.158797 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:42:37.159168 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:42:37.159177 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:42:37.159178 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:42:37.159357 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:42:37.159799 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:42:37.159808 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:42:37.160093 (+ 285us) client_negotiation.cc:769] Sending connection context
0925 22:42:37.160306 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 22:42:37.160676 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":155,"threads_started":1}
I20260925 22:42:37.163177 14047 heartbeater.cc:347] Connected to a master server at 127.2.74.254:35039
I20260925 22:42:37.163473 14047 heartbeater.cc:464] Registering TS with master...
I20260925 22:42:37.164055 14047 heartbeater.cc:511] Master 127.2.74.254:35039 requested a full tablet report, sending...
I20260925 22:42:37.165369 13563 ts_manager.cc:194] Registered new tserver with Master: 6db2367a328d4ba6b59ec84f5bdfa3f6 (127.2.74.195:36585)
I20260925 22:42:37.166082 13563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.195@KRBTEST.COM'} at 127.2.74.195:53555
I20260925 22:42:37.170369 2347 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:42:37.182281 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:42:37.172657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50660 (local address 127.2.74.254:35039)
0925 22:42:37.172780 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 22:42:37.172784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:42:37.173446 (+ 662us) server_negotiation.cc:407] Connection header received
0925 22:42:37.173607 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:42:37.173610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:42:37.173665 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:42:37.173754 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:42:37.174493 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:42:37.175198 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:42:37.176332 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:42:37.176656 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:42:37.176730 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:42:37.178628 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:42:37.178632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:42:37.178635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:42:37.178666 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:42:37.180402 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:42:37.180928 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:42:37.180932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:42:37.180933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:42:37.181019 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:42:37.181304 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:42:37.181307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:42:37.181309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:42:37.181517 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:42:37.181636 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 22:42:37.181989 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 22:42:37.182115 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:42:37.186775 13563 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:42:37.191556 13563 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:42:37.209816 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:42:37.203047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:40537 (local address 127.0.0.1:50448)
0925 22:42:37.203498 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 22:42:37.203554 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 22:42:37.203696 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:42:37.204885 (+ 1189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:42:37.204891 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:42:37.204916 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:42:37.205101 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:42:37.205108 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:42:37.206331 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:42:37.206334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:42:37.207655 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:42:37.207666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:42:37.208796 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:42:37.208801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:42:37.209368 (+ 567us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:42:37.209424 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 22:42:37.209528 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 22:42:37.209651 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":4,"thread_start_us":136,"threads_started":1}
I20260925 22:42:37.211788 14062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:42:37.204035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50448 (local address 127.2.74.193:40537)
0925 22:42:37.204331 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0925 22:42:37.204341 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:42:37.204367 (+ 26us) server_negotiation.cc:407] Connection header received
0925 22:42:37.204451 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:42:37.204466 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:42:37.204704 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:42:37.204884 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:42:37.205234 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:42:37.206189 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:42:37.207808 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:42:37.208659 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:42:37.211298 (+ 2639us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:42:37.211503 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 22:42:37.211563 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 22:42:37.211665 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":110,"threads_started":1}
I20260925 22:42:37.213575 14060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:42:37.204247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:36585 (local address 127.0.0.1:45618)
0925 22:42:37.207772 (+ 3525us) negotiation.cc:107] Waiting for socket to connect
0925 22:42:37.207787 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:42:37.207893 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:42:37.208601 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:42:37.208605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:42:37.208624 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:42:37.208742 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:42:37.208752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:42:37.210001 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:42:37.210004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:42:37.211397 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:42:37.211408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:42:37.213142 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:42:37.213145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:42:37.213221 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:42:37.213241 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 22:42:37.213345 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 22:42:37.213425 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3455,"mutex_wait_us":1,"thread_start_us":98,"threads_started":1}
I20260925 22:42:37.213624 14064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:42:37.204660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45618 (local address 127.2.74.195:36585)
0925 22:42:37.205723 (+ 1063us) server_negotiation.cc:206] Beginning negotiation
0925 22:42:37.205739 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:42:37.207878 (+ 2139us) server_negotiation.cc:407] Connection header received
0925 22:42:37.208023 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:42:37.208030 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:42:37.208275 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:42:37.208443 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:42:37.208872 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:42:37.209854 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:42:37.212010 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:42:37.212817 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:42:37.212928 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:42:37.213086 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 22:42:37.213380 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 22:42:37.213490 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":932,"thread_start_us":141,"threads_started":1}
I20260925 22:42:37.214197 14063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:42:37.203135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:33377 (local address 127.0.0.1:50224)
0925 22:42:37.205295 (+ 2160us) negotiation.cc:107] Waiting for socket to connect
0925 22:42:37.205317 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 22:42:37.205423 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:42:37.208732 (+ 3309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:42:37.208735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:42:37.208756 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:42:37.209347 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:42:37.209359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:42:37.210919 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:42:37.210922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:42:37.212641 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:42:37.212653 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:42:37.213818 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:42:37.213822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:42:37.213915 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:42:37.213938 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:42:37.213997 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 22:42:37.214068 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2089,"mutex_wait_us":463,"thread_start_us":100,"threads_started":1}
I20260925 22:42:37.214875 14061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:42:37.203276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50224 (local address 127.2.74.194:33377)
0925 22:42:37.207507 (+ 4231us) server_negotiation.cc:206] Beginning negotiation
0925 22:42:37.208012 (+ 505us) server_negotiation.cc:399] Waiting for connection header
0925 22:42:37.208043 (+ 31us) server_negotiation.cc:407] Connection header received
0925 22:42:37.208136 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:42:37.208142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:42:37.208376 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:42:37.208554 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:42:37.209501 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:42:37.210767 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:42:37.212809 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:42:37.213578 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:42:37.213684 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:42:37.214590 (+ 906us) server_negotiation.cc:1035] Waiting for connection context
0925 22:42:37.214643 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 22:42:37.214738 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4096,"thread_start_us":82,"threads_started":1}
I20260925 22:42:37.215387 13709 tablet_service.cc:1467] Processing CreateTablet for tablet 1c5f1b567e5b41cfa3ce589b08dfb78a (DEFAULT_TABLE table=test-table [id=437feb340cc14e209f97f472795a308d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:42:37.216398 13709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c5f1b567e5b41cfa3ce589b08dfb78a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:42:37.216516 13977 tablet_service.cc:1467] Processing CreateTablet for tablet 1c5f1b567e5b41cfa3ce589b08dfb78a (DEFAULT_TABLE table=test-table [id=437feb340cc14e209f97f472795a308d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:42:37.217370 13977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c5f1b567e5b41cfa3ce589b08dfb78a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:42:37.221781 13846 tablet_service.cc:1467] Processing CreateTablet for tablet 1c5f1b567e5b41cfa3ce589b08dfb78a (DEFAULT_TABLE table=test-table [id=437feb340cc14e209f97f472795a308d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:42:37.223107 13846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c5f1b567e5b41cfa3ce589b08dfb78a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:42:37.225431 14065 tablet_bootstrap.cc:492] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 5bc4a646adb74c05ba923c9939e46643: Bootstrap starting.
I20260925 22:42:37.228058 14065 tablet_bootstrap.cc:654] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 5bc4a646adb74c05ba923c9939e46643: Neither blocks nor log segments found. Creating new log.
I20260925 22:42:37.228981 14065 log.cc:824] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 5bc4a646adb74c05ba923c9939e46643: Log is configured to *not* fsync() on all Append() calls
I20260925 22:42:37.231220 14068 tablet_bootstrap.cc:492] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 9f6fce6b8c8b4506b17475cd495021b0: Bootstrap starting.
I20260925 22:42:37.232607 14066 tablet_bootstrap.cc:492] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 6db2367a328d4ba6b59ec84f5bdfa3f6: Bootstrap starting.
I20260925 22:42:37.234658 14068 tablet_bootstrap.cc:654] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 9f6fce6b8c8b4506b17475cd495021b0: Neither blocks nor log segments found. Creating new log.
I20260925 22:42:37.235678 14068 log.cc:824] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 9f6fce6b8c8b4506b17475cd495021b0: Log is configured to *not* fsync() on all Append() calls
I20260925 22:42:37.236937 14065 tablet_bootstrap.cc:492] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 5bc4a646adb74c05ba923c9939e46643: No bootstrap required, opened a new log
I20260925 22:42:37.237279 14065 ts_tablet_manager.cc:1397] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 5bc4a646adb74c05ba923c9939e46643: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260925 22:42:37.238061 14068 tablet_bootstrap.cc:492] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 9f6fce6b8c8b4506b17475cd495021b0: No bootstrap required, opened a new log
I20260925 22:42:37.238265 14068 ts_tablet_manager.cc:1397] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 9f6fce6b8c8b4506b17475cd495021b0: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260925 22:42:37.242779 14065 raft_consensus.cc:348] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 5bc4a646adb74c05ba923c9939e46643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bc4a646adb74c05ba923c9939e46643" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40537 } } peers { permanent_uuid: "9f6fce6b8c8b4506b17475cd495021b0" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 33377 } } peers { permanent_uuid: "6db2367a328d4ba6b59ec84f5bdfa3f6" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36585 } }
I20260925 22:42:37.243216 14068 raft_consensus.cc:348] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 9f6fce6b8c8b4506b17475cd495021b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bc4a646adb74c05ba923c9939e46643" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40537 } } peers { permanent_uuid: "9f6fce6b8c8b4506b17475cd495021b0" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 33377 } } peers { permanent_uuid: "6db2367a328d4ba6b59ec84f5bdfa3f6" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36585 } }
I20260925 22:42:37.243489 14068 raft_consensus.cc:374] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 9f6fce6b8c8b4506b17475cd495021b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:42:37.243559 14068 raft_consensus.cc:729] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 9f6fce6b8c8b4506b17475cd495021b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6fce6b8c8b4506b17475cd495021b0, State: Initialized, Role: FOLLOWER
I20260925 22:42:37.243808 14065 raft_consensus.cc:374] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 5bc4a646adb74c05ba923c9939e46643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:42:37.244055 14065 raft_consensus.cc:729] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 5bc4a646adb74c05ba923c9939e46643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bc4a646adb74c05ba923c9939e46643, State: Initialized, Role: FOLLOWER
I20260925 22:42:37.244244 14068 consensus_queue.cc:260] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 9f6fce6b8c8b4506b17475cd495021b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bc4a646adb74c05ba923c9939e46643" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40537 } } peers { permanent_uuid: "9f6fce6b8c8b4506b17475cd495021b0" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 33377 } } peers { permanent_uuid: "6db2367a328d4ba6b59ec84f5bdfa3f6" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36585 } }
I20260925 22:42:37.244765 14065 consensus_queue.cc:260] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 5bc4a646adb74c05ba923c9939e46643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bc4a646adb74c05ba923c9939e46643" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40537 } } peers { permanent_uuid: "9f6fce6b8c8b4506b17475cd495021b0" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 33377 } } peers { permanent_uuid: "6db2367a328d4ba6b59ec84f5bdfa3f6" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36585 } }
I20260925 22:42:37.245530 14068 ts_tablet_manager.cc:1428] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 9f6fce6b8c8b4506b17475cd495021b0: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260925 22:42:37.245967 13913 heartbeater.cc:503] Master 127.2.74.254:35039 was elected leader, sending a full tablet report...
I20260925 22:42:37.246081 14065 ts_tablet_manager.cc:1428] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 5bc4a646adb74c05ba923c9939e46643: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260925 22:42:37.246464 13775 heartbeater.cc:503] Master 127.2.74.254:35039 was elected leader, sending a full tablet report...
I20260925 22:42:37.249738 14066 tablet_bootstrap.cc:654] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 6db2367a328d4ba6b59ec84f5bdfa3f6: Neither blocks nor log segments found. Creating new log.
I20260925 22:42:37.250893 14066 log.cc:824] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 6db2367a328d4ba6b59ec84f5bdfa3f6: Log is configured to *not* fsync() on all Append() calls
I20260925 22:42:37.254837 14066 tablet_bootstrap.cc:492] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 6db2367a328d4ba6b59ec84f5bdfa3f6: No bootstrap required, opened a new log
I20260925 22:42:37.255119 14066 ts_tablet_manager.cc:1397] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 6db2367a328d4ba6b59ec84f5bdfa3f6: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.000s
I20260925 22:42:37.258380 14066 raft_consensus.cc:348] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 6db2367a328d4ba6b59ec84f5bdfa3f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bc4a646adb74c05ba923c9939e46643" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40537 } } peers { permanent_uuid: "9f6fce6b8c8b4506b17475cd495021b0" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 33377 } } peers { permanent_uuid: "6db2367a328d4ba6b59ec84f5bdfa3f6" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36585 } }
I20260925 22:42:37.258702 14066 raft_consensus.cc:374] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 6db2367a328d4ba6b59ec84f5bdfa3f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:42:37.258764 14066 raft_consensus.cc:729] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 6db2367a328d4ba6b59ec84f5bdfa3f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6db2367a328d4ba6b59ec84f5bdfa3f6, State: Initialized, Role: FOLLOWER
I20260925 22:42:37.259349 14066 consensus_queue.cc:260] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 6db2367a328d4ba6b59ec84f5bdfa3f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bc4a646adb74c05ba923c9939e46643" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40537 } } peers { permanent_uuid: "9f6fce6b8c8b4506b17475cd495021b0" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 33377 } } peers { permanent_uuid: "6db2367a328d4ba6b59ec84f5bdfa3f6" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36585 } }
I20260925 22:42:37.260145 14047 heartbeater.cc:503] Master 127.2.74.254:35039 was elected leader, sending a full tablet report...
I20260925 22:42:37.260564 14066 ts_tablet_manager.cc:1428] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 6db2367a328d4ba6b59ec84f5bdfa3f6: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
Sep 25 22:42:37 dist-test-slave-csnl krb5kdc[13087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376156, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for HTTP/127.2.74.212@KRBTEST.COM
W20260925 22:42:37.397584 14048 tablet.cc:2369] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 6db2367a328d4ba6b59ec84f5bdfa3f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:42:37.412390 14071 raft_consensus.cc:482] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 5bc4a646adb74c05ba923c9939e46643 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:42:37.412659 14071 raft_consensus.cc:504] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 5bc4a646adb74c05ba923c9939e46643 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bc4a646adb74c05ba923c9939e46643" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40537 } } peers { permanent_uuid: "9f6fce6b8c8b4506b17475cd495021b0" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 33377 } } peers { permanent_uuid: "6db2367a328d4ba6b59ec84f5bdfa3f6" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36585 } }
I20260925 22:42:37.414028 14071 leader_election.cc:288] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 5bc4a646adb74c05ba923c9939e46643 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f6fce6b8c8b4506b17475cd495021b0 (127.2.74.194:33377), 6db2367a328d4ba6b59ec84f5bdfa3f6 (127.2.74.195:36585)
I20260925 22:42:37.421490 14064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:42:37.415331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:58363 (local address 127.2.74.195:36585)
0925 22:42:37.415457 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:42:37.415462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:42:37.416686 (+ 1224us) server_negotiation.cc:407] Connection header received
0925 22:42:37.416898 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:42:37.416904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:42:37.416963 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:42:37.417012 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:42:37.417494 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:42:37.418224 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:42:37.419983 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:42:37.420624 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:42:37.420731 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:42:37.420788 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 22:42:37.421231 (+ 443us) server_negotiation.cc:299] Negotiation successful
0925 22:42:37.421329 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:42:37.422223 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:42:37.414851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:36585 (local address 127.2.74.193:58363)
0925 22:42:37.416599 (+ 1748us) negotiation.cc:107] Waiting for socket to connect
0925 22:42:37.416615 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:42:37.416764 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:42:37.417159 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:42:37.417162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:42:37.417180 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:42:37.417339 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:42:37.417347 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:42:37.418407 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:42:37.418412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:42:37.419821 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:42:37.419839 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:42:37.420955 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:42:37.420958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:42:37.421060 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:42:37.421098 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 22:42:37.421963 (+ 865us) client_negotiation.cc:240] Negotiation successful
0925 22:42:37.422062 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1661,"thread_start_us":800,"threads_started":1}
I20260925 22:42:37.423202 13983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5f1b567e5b41cfa3ce589b08dfb78a" candidate_uuid: "5bc4a646adb74c05ba923c9939e46643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6db2367a328d4ba6b59ec84f5bdfa3f6" is_pre_election: true
I20260925 22:42:37.423588 13983 raft_consensus.cc:2454] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 6db2367a328d4ba6b59ec84f5bdfa3f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bc4a646adb74c05ba923c9939e46643 in term 0.
I20260925 22:42:37.424317 13664 leader_election.cc:302] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 5bc4a646adb74c05ba923c9939e46643 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bc4a646adb74c05ba923c9939e46643, 6db2367a328d4ba6b59ec84f5bdfa3f6; no voters:
I20260925 22:42:37.424880 14071 raft_consensus.cc:2781] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 5bc4a646adb74c05ba923c9939e46643 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:42:37.424986 14071 raft_consensus.cc:482] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 5bc4a646adb74c05ba923c9939e46643 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:42:37.425033 14071 raft_consensus.cc:3037] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 5bc4a646adb74c05ba923c9939e46643 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:42:37.426545 14071 raft_consensus.cc:504] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 5bc4a646adb74c05ba923c9939e46643 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bc4a646adb74c05ba923c9939e46643" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40537 } } peers { permanent_uuid: "9f6fce6b8c8b4506b17475cd495021b0" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 33377 } } peers { permanent_uuid: "6db2367a328d4ba6b59ec84f5bdfa3f6" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36585 } }
I20260925 22:42:37.427315 14071 leader_election.cc:288] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 5bc4a646adb74c05ba923c9939e46643 [CANDIDATE]: Term 1 election: Requested vote from peers 9f6fce6b8c8b4506b17475cd495021b0 (127.2.74.194:33377), 6db2367a328d4ba6b59ec84f5bdfa3f6 (127.2.74.195:36585)
I20260925 22:42:37.427915 13983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5f1b567e5b41cfa3ce589b08dfb78a" candidate_uuid: "5bc4a646adb74c05ba923c9939e46643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6db2367a328d4ba6b59ec84f5bdfa3f6"
I20260925 22:42:37.428074 13983 raft_consensus.cc:3037] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 6db2367a328d4ba6b59ec84f5bdfa3f6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:42:37.429569 13983 raft_consensus.cc:2454] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 6db2367a328d4ba6b59ec84f5bdfa3f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bc4a646adb74c05ba923c9939e46643 in term 1.
W20260925 22:42:37.428555 13776 tablet.cc:2369] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 5bc4a646adb74c05ba923c9939e46643: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:42:37.430109 13664 leader_election.cc:302] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 5bc4a646adb74c05ba923c9939e46643 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bc4a646adb74c05ba923c9939e46643, 6db2367a328d4ba6b59ec84f5bdfa3f6; no voters:
I20260925 22:42:37.431102 14071 raft_consensus.cc:2781] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 5bc4a646adb74c05ba923c9939e46643 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:42:37.431239 14071 raft_consensus.cc:686] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 5bc4a646adb74c05ba923c9939e46643 [term 1 LEADER]: Becoming Leader. State: Replica: 5bc4a646adb74c05ba923c9939e46643, State: Running, Role: LEADER
I20260925 22:42:37.431351 14061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:42:37.415089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:34385 (local address 127.2.74.194:33377)
0925 22:42:37.415316 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 22:42:37.415321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:42:37.415336 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:42:37.415391 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:42:37.415394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:42:37.415441 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:42:37.415519 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:42:37.423390 (+ 7871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:42:37.424154 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:42:37.425675 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:42:37.426281 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:42:37.426386 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:42:37.426446 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 22:42:37.431089 (+ 4643us) server_negotiation.cc:299] Negotiation successful
0925 22:42:37.431218 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260925 22:42:37.431841 14075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:42:37.414418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:33377 (local address 127.2.74.193:34385)
0925 22:42:37.414791 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0925 22:42:37.414809 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:42:37.414921 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:42:37.423042 (+ 8121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:42:37.423046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:42:37.423080 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:42:37.423217 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:42:37.423224 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:42:37.424320 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:42:37.424323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:42:37.425494 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:42:37.425506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:42:37.426556 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:42:37.426559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:42:37.426658 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:42:37.426683 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:42:37.431582 (+ 4899us) client_negotiation.cc:240] Negotiation successful
0925 22:42:37.431685 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":143,"threads_started":1}
I20260925 22:42:37.431821 14071 consensus_queue.cc:237] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 5bc4a646adb74c05ba923c9939e46643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bc4a646adb74c05ba923c9939e46643" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40537 } } peers { permanent_uuid: "9f6fce6b8c8b4506b17475cd495021b0" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 33377 } } peers { permanent_uuid: "6db2367a328d4ba6b59ec84f5bdfa3f6" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36585 } }
I20260925 22:42:37.432624 13852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5f1b567e5b41cfa3ce589b08dfb78a" candidate_uuid: "5bc4a646adb74c05ba923c9939e46643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6fce6b8c8b4506b17475cd495021b0" is_pre_election: true
I20260925 22:42:37.432940 13852 raft_consensus.cc:2454] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 9f6fce6b8c8b4506b17475cd495021b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bc4a646adb74c05ba923c9939e46643 in term 0.
I20260925 22:42:37.433378 13853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5f1b567e5b41cfa3ce589b08dfb78a" candidate_uuid: "5bc4a646adb74c05ba923c9939e46643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6fce6b8c8b4506b17475cd495021b0"
I20260925 22:42:37.433492 13853 raft_consensus.cc:3037] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 9f6fce6b8c8b4506b17475cd495021b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:42:37.434876 13853 raft_consensus.cc:2454] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 9f6fce6b8c8b4506b17475cd495021b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bc4a646adb74c05ba923c9939e46643 in term 1.
I20260925 22:42:37.437757 13562 catalog_manager.cc:5495] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 5bc4a646adb74c05ba923c9939e46643 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bc4a646adb74c05ba923c9939e46643 (127.2.74.193). New cstate: current_term: 1 leader_uuid: "5bc4a646adb74c05ba923c9939e46643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bc4a646adb74c05ba923c9939e46643" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 40537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f6fce6b8c8b4506b17475cd495021b0" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 33377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6db2367a328d4ba6b59ec84f5bdfa3f6" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36585 } health_report { overall_health: UNKNOWN } } }
W20260925 22:42:37.491833 13914 tablet.cc:2369] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 9f6fce6b8c8b4506b17475cd495021b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:42:37.509119 14062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:42:37.503770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38912 (local address 127.2.74.193:40537)
0925 22:42:37.503915 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:42:37.503920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:42:37.504050 (+ 130us) server_negotiation.cc:407] Connection header received
0925 22:42:37.504238 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:42:37.504242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:42:37.504320 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:42:37.504444 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:42:37.504730 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:42:37.505508 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:42:37.506822 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:42:37.507190 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:42:37.507296 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:42:37.507776 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:42:37.508377 (+ 601us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:42:37.508511 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 22:42:37.508889 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 22:42:37.508960 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 22:42:37.516464 13983 raft_consensus.cc:1261] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 6db2367a328d4ba6b59ec84f5bdfa3f6 [term 1 FOLLOWER]: Refusing update from remote peer 5bc4a646adb74c05ba923c9939e46643: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:42:37.517429 14071 consensus_queue.cc:1035] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 5bc4a646adb74c05ba923c9939e46643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6db2367a328d4ba6b59ec84f5bdfa3f6" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:42:37.519008 13853 raft_consensus.cc:1261] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 9f6fce6b8c8b4506b17475cd495021b0 [term 1 FOLLOWER]: Refusing update from remote peer 5bc4a646adb74c05ba923c9939e46643: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:42:37.519768 14077 consensus_queue.cc:1035] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 5bc4a646adb74c05ba923c9939e46643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f6fce6b8c8b4506b17475cd495021b0" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 33377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:42:37.530570 14081 mvcc.cc:205] Tried to move back new op lower bound from 7333380741178937344 to 7333380740850049024. Current Snapshot: MvccSnapshot[applied={T|T < 7333380741178937344}]
I20260925 22:42:37.536222 14084 mvcc.cc:205] Tried to move back new op lower bound from 7333380741178937344 to 7333380740850049024. Current Snapshot: MvccSnapshot[applied={T|T < 7333380741178937344}]
I20260925 22:42:37.537623 14082 mvcc.cc:205] Tried to move back new op lower bound from 7333380741178937344 to 7333380740850049024. Current Snapshot: MvccSnapshot[applied={T|T < 7333380741178937344}]
I20260925 22:42:37.547706 13563 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:42:37.547931 13563 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:42:37.551026 13563 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8bc436dd21774e54b3efc0e181377bb2: Sending DeleteTablet for 3 replicas of tablet 1c5f1b567e5b41cfa3ce589b08dfb78a
I20260925 22:42:37.552517 13846 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c5f1b567e5b41cfa3ce589b08dfb78a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:42:37 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:50224
I20260925 22:42:37.553145 13977 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c5f1b567e5b41cfa3ce589b08dfb78a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:42:37 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:45618
I20260925 22:42:37.554054 13709 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c5f1b567e5b41cfa3ce589b08dfb78a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:42:37 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:50448
I20260925 22:42:37.555353 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 13644
I20260925 22:42:37.571338 14090 tablet_replica.cc:330] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 9f6fce6b8c8b4506b17475cd495021b0: stopping tablet replica
I20260925 22:42:37.573328 14091 tablet_replica.cc:330] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 6db2367a328d4ba6b59ec84f5bdfa3f6: stopping tablet replica
I20260925 22:42:37.573766 14091 raft_consensus.cc:2229] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 6db2367a328d4ba6b59ec84f5bdfa3f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:42:37.574184 14091 raft_consensus.cc:2258] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 6db2367a328d4ba6b59ec84f5bdfa3f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:42:37.574728 14090 raft_consensus.cc:2229] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 9f6fce6b8c8b4506b17475cd495021b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:42:37.575178 14090 raft_consensus.cc:2258] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 9f6fce6b8c8b4506b17475cd495021b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:42:37.577148 14090 ts_tablet_manager.cc:1905] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 9f6fce6b8c8b4506b17475cd495021b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:42:37.577168 14091 ts_tablet_manager.cc:1905] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 6db2367a328d4ba6b59ec84f5bdfa3f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:42:37.580397 14091 ts_tablet_manager.cc:1918] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 6db2367a328d4ba6b59ec84f5bdfa3f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:42:37.580535 14091 log.cc:1196] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 6db2367a328d4ba6b59ec84f5bdfa3f6: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-2/wal/wals/1c5f1b567e5b41cfa3ce589b08dfb78a
I20260925 22:42:37.580662 14090 ts_tablet_manager.cc:1918] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 9f6fce6b8c8b4506b17475cd495021b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:42:37.580875 14090 log.cc:1196] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 9f6fce6b8c8b4506b17475cd495021b0: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376047580784-2347-0/minicluster-data/ts-1/wal/wals/1c5f1b567e5b41cfa3ce589b08dfb78a
I20260925 22:42:37.580888 14091 ts_tablet_manager.cc:1939] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 6db2367a328d4ba6b59ec84f5bdfa3f6: Deleting consensus metadata
I20260925 22:42:37.581281 14090 ts_tablet_manager.cc:1939] T 1c5f1b567e5b41cfa3ce589b08dfb78a P 9f6fce6b8c8b4506b17475cd495021b0: Deleting consensus metadata
I20260925 22:42:37.582420 13549 catalog_manager.cc:4841] TS 6db2367a328d4ba6b59ec84f5bdfa3f6 (127.2.74.195:36585): tablet 1c5f1b567e5b41cfa3ce589b08dfb78a (table test-table [id=437feb340cc14e209f97f472795a308d]) successfully deleted
I20260925 22:42:37.582705 13549 catalog_manager.cc:4841] TS 9f6fce6b8c8b4506b17475cd495021b0 (127.2.74.194:33377): tablet 1c5f1b567e5b41cfa3ce589b08dfb78a (table test-table [id=437feb340cc14e209f97f472795a308d]) successfully deleted
W20260925 22:42:37.583483 13550 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.74.193:40537 (error 108)
I20260925 22:42:37.584589 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 13783
I20260925 22:42:37.584630 14063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:42:37.584316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:40537 (local address 127.0.0.1:38914)
0925 22:42:37.584447 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 22:42:37.584522 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.193:40537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 22:42:37.586125 13550 catalog_manager.cc:4584] TS 5bc4a646adb74c05ba923c9939e46643 (127.2.74.193:40537): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c5f1b567e5b41cfa3ce589b08dfb78a: Network error: Client connection negotiation failed: client connection to 127.2.74.193:40537: connect: Connection refused (error 111)
I20260925 22:42:37.591720 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 13918
I20260925 22:42:37.601281 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 13531
I20260925 22:42:37.608361 2347 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 22:42:37.944327 2347 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 22:42:37.944450 2347 mini_ranger.cc:66] Stopping Ranger...
I20260925 22:42:38.290200 2347 mini_ranger.cc:68] Stopped Ranger
2026-09-25 22:42:38.290 UTC [13117] LOG: received smart shutdown request
2026-09-25 22:42:38.294 UTC [13117] LOG: background worker "logical replication launcher" (PID 13126) exited with exit code 1
2026-09-25 22:42:38.294 UTC [13121] LOG: shutting down
2026-09-25 22:42:38.473 UTC [13117] LOG: database system is shut down
2026-09-25T22:42:38Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48362 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:42:38 dist-test-slave-csnl krb5kdc[14102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:42:38 dist-test-slave-csnl krb5kdc[14102](info): set up 2 sockets
Sep 25 22:42:38 dist-test-slave-csnl krb5kdc[14102](info): commencing operation
krb5kdc: starting...
W20260925 22:42:39.663075 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:42:39 dist-test-slave-csnl krb5kdc[14102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:42:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:42:39Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.74.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.74.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.74.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/rangeradmin_127.2.74.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.74.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/rangeradmin_127.2.74.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.74.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.74.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.74.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/rangerlookup_127.2.74.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.74.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/rangerlookup_127.2.74.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.74.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.74.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.74.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/HTTP_127.2.74.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.74.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/HTTP_127.2.74.212@KRBTEST.COM.keytab.
I20260925 22:42:39.834095 2347 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskx4wBtU/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/postgres -l logfile start
2026-09-25 22:42:42.351 UTC [14132] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 22:42:42.351 UTC [14132] LOG: listening on IPv4 address "127.2.74.212", port 56015
2026-09-25 22:42:42.355 UTC [14132] LOG: listening on Unix socket "/tmp/.s.PGSQL.56015"
2026-09-25 22:42:42.377 UTC [14135] LOG: database system was shut down at 2026-09-25 22:42:40 UTC
2026-09-25 22:42:42.381 UTC [14132] LOG: database system is ready to accept connections
I20260925 22:42:43.350904 2347 mini_postgres.cc:96] Postgres bound to 56015
2026-09-25 22:42:43.357 UTC [14143] FATAL: database "slave" does not exist
127.2.74.212:56015 - accepting connections
I20260925 22:42:43.358284 2347 mini_ranger.cc:161] Starting Ranger...
I20260925 22:42:43.375044 2347 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 22:42:43.948565 2347 mini_ranger.cc:87] Created ranger Postgres database
I20260925 22:42:43.948671 2347 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ranger-admin
2026-09-25 22:42:44,226 [I] DB FLAVOR :POSTGRES
2026-09-25 22:42:44,226 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 22:42:44,226 [I] Checking connection..
2026-09-25 22:42:44,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:56015/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 22:42:44,495 [I] Checking connection passed.
2026-09-25 22:42:44,495 [I] --------- Verifying version history table ---------
2026-09-25 22:42:44,495 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:56015/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:42:44,746 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 22:42:44,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:56015/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:42:45,010 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 22:42:45,010 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 22:42:45,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:56015/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 22:42:45.262 UTC [14207] WARNING: there is no transaction in progress
2026-09-25 22:42:45,269 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 22:42:45,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:56015/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:42:45,515 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 22:42:45,515 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 22:42:45,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:56015/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 22:42:45,755 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:56015/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 22:42:46,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:56015/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-csnl.c.gcp-upstream.internal','N') ;"
2026-09-25 22:42:46,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:56015/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 22:42:46,485 [I] Table x_portal_user does not exist in database ranger
2026-09-25 22:42:46,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:56015/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 22:42:46,731 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 22:42:46,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:56015/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 22:42:46,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:56015/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 22:42:47,247 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 22:42:47,247 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:56015/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 22:42:48.277 UTC [14315] WARNING: there is no transaction in progress
2026-09-25 22:42:48.294 UTC [14315] WARNING: there is no transaction in progress
2026-09-25 22:42:48.312 UTC [14315] WARNING: there is no transaction in progress
2026-09-25 22:42:48.330 UTC [14315] WARNING: there is no transaction in progress
2026-09-25 22:42:48.348 UTC [14315] WARNING: there is no transaction in progress
2026-09-25 22:42:48.368 UTC [14315] WARNING: there is no transaction in progress
2026-09-25 22:42:48.436 UTC [14315] WARNING: there is no transaction in progress
2026-09-25 22:42:48.444 UTC [14315] WARNING: there is no transaction in progress
2026-09-25 22:42:48.463 UTC [14315] WARNING: there is no transaction in progress
2026-09-25 22:42:48.478 UTC [14315] WARNING: there is no transaction in progress
2026-09-25 22:42:48.492 UTC [14315] WARNING: there is no transaction in progress
2026-09-25 22:42:48.508 UTC [14315] WARNING: there is no transaction in progress
2026-09-25 22:42:48.519 UTC [14315] WARNING: there is no transaction in progress
2026-09-25 22:42:48.527 UTC [14315] WARNING: there is no transaction in progress
2026-09-25 22:42:48.536 UTC [14315] WARNING: there is no transaction in progress
2026-09-25 22:42:48.544 UTC [14315] WARNING: there is no transaction in progress
2026-09-25 22:42:49,285 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 22:42:49,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:56015/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 22:42:49,553 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 22:42:49,554 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:56015/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 22:42:49,799 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 22:42:49,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:56015/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 22:42:50,079 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 22:42:50,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:56015/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 22:42:50,342 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:56015/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 22:42:50,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:56015/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-csnl.c.gcp-upstream.internal';"
2026-09-25 22:42:50,846 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 22:42:50,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:56015/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 22:42:51,123 [I] DB_PATCHES have already been applied
I20260925 22:42:51.129467 2347 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ranger-admin:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskx4wBtU/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 22:42:51.129537 2347 mini_ranger.cc:193] Using host: 127.2.74.212
I20260925 22:42:51.133072 2347 mini_ranger.cc:227] Ranger admin URL: http://127.2.74.212:50339
Sep 25, 2026 10:42:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:42:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:42:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 10:42:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 10:42:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 10:42:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 10:42:52 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 22:42:52.252758 2347 mini_ranger.cc:160] Time spent starting Ranger: real 8.894s user 0.000s sys 0.010s
Sep 25, 2026 10:43:14 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 10:43:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 10:43:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 10:43:14 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 10:43:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 10:43:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 10:43:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 10:43:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 10:43:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 10:43:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 10:43:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 10:43:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 10:43:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 10:43:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 10:43:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 10:43:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 10:43:14 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 10:43:15 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 22:43:15.883849 2347 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.74.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.74.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.74.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/rangerkms_127.2.74.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.74.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/rangerkms_127.2.74.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.74.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.74.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.74.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/HTTP_127.2.74.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.74.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/HTTP_127.2.74.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 22:43:15 dist-test-slave-csnl krb5kdc[14102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376195, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 22:43:15.974825 2347 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 22:43:15.985116 2347 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 22:43:16.901670 2347 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 22:43:16.901769 2347 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ranger-kms
I20260925 22:43:16.901793 2347 mini_ranger_kms.cc:227] Using postgres at 127.2.74.212:56015
2026-09-25 22:43:17,077 [I] DB FLAVOR :POSTGRES
2026-09-25 22:43:17,077 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 22:43:17,077 [I] Checking connection
2026-09-25 22:43:17,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:56015/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 22:43:17,347 [I] connection success
2026-09-25 22:43:17,347 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 22:43:17,347 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 22:43:17,348 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:56015/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 22:43:17,596 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 22:43:17,597 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 22:43:17,597 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 22:43:17,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.74.212:56015/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 22:43:17,897 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 22:43:18.436950 2347 mini_ranger_kms.cc:310] Created kms service
I20260925 22:43:18.625865 2347 mini_ranger_kms.cc:326] Created kudu user
I20260925 22:43:18.676852 2347 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 22:43:18 dist-test-slave-csnl krb5kdc[14102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376195, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.74.212@KRBTEST.COM
I20260925 22:43:19.148514 2347 mini_ranger_kms.cc:384] Added ranger policy
I20260925 22:43:19.148679 2347 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ranger-kms:/tmp/dist-test-taskx4wBtU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskx4wBtU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 22:43:19.148701 2347 mini_ranger_kms.cc:240] Using host: 127.2.74.212
I20260925 22:43:19.153144 2347 mini_ranger_kms.cc:276] Ranger KMS PID: 14496
I20260925 22:43:19.153278 2347 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.74.212:52161
Sep 25, 2026 10:43:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:43:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:43:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 10:43:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 10:43:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 10:43:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 22:43:21.406873 2347 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.432s user 0.023s sys 0.134s
I20260925 22:43:21.407034 2347 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 22:43:21.407106 2347 mini_ranger_kms.cc:401] 127.2.74.212:52161/kms/v1/keys
Sep 25 22:43:21 dist-test-slave-csnl krb5kdc[14102](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790376201, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.74.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 22:43:22 dist-test-slave-csnl krb5kdc[14102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376201, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.74.212@KRBTEST.COM for HTTP/127.2.74.212@KRBTEST.COM
Sep 25 22:43:22 dist-test-slave-csnl krb5kdc[14102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376201, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.74.212@KRBTEST.COM for HTTP/127.2.74.212@KRBTEST.COM
Sep 25 22:43:24 dist-test-slave-csnl krb5kdc[14102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:43:24.791564 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:38849
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:38849
--ranger_config_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.74.212:52161/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:43:24.934669 14546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:43:24.935050 14546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:43:24.935112 14546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:43:24.939452 14546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:43:24.939528 14546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:43:24.939548 14546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:43:24.939565 14546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:43:24.939581 14546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:43:24.945609 14546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.74.212:52161/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:38849
--ranger_config_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:38849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:43:24.946749 14546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:43:24.947901 14546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:43:24.956806 14552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:24.956786 14554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:24.956768 14551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:24.957170 14546 server_base.cc:1034] running on GCE node
I20260925 22:43:24.957702 14546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:43:24.958875 14546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:43:24.960060 14546 hybrid_clock.cc:648] HybridClock initialized: now 1790376204960033 us; error 41 us; skew 500 ppm
Sep 25 22:43:24 dist-test-slave-csnl krb5kdc[14102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376204, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:43:24.969015 14546 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:43:24.970034 14546 webserver.cc:460] Webserver started at http://127.2.74.254:35263/ using document root <none> and password file <none>
I20260925 22:43:24.970613 14546 fs_manager.cc:359] Metadata directory not provided
I20260925 22:43:24.970698 14546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:43:24.970927 14546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:43:24 dist-test-slave-csnl krb5kdc[14102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376204, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for HTTP/127.2.74.212@KRBTEST.COM
I20260925 22:43:25.029781 14546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "110b9b188a944147953c1a74fe93cb9e"
format_stamp: "Formatted at 2026-09-25 22:43:25 on dist-test-slave-csnl"
server_key: "be4010ea295f90b48c112bf9132d7aee"
server_key_iv: "ff0553a1248b59d2ffd186de4f413aa2"
server_key_version: "kuduclusterkey@0"
I20260925 22:43:25.030499 14546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "110b9b188a944147953c1a74fe93cb9e"
format_stamp: "Formatted at 2026-09-25 22:43:25 on dist-test-slave-csnl"
server_key: "be4010ea295f90b48c112bf9132d7aee"
server_key_iv: "ff0553a1248b59d2ffd186de4f413aa2"
server_key_version: "kuduclusterkey@0"
I20260925 22:43:25.034811 14546 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:43:25.053181 14561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:25.054695 14546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:43:25.054872 14546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "110b9b188a944147953c1a74fe93cb9e"
format_stamp: "Formatted at 2026-09-25 22:43:25 on dist-test-slave-csnl"
server_key: "be4010ea295f90b48c112bf9132d7aee"
server_key_iv: "ff0553a1248b59d2ffd186de4f413aa2"
server_key_version: "kuduclusterkey@0"
I20260925 22:43:25.055066 14546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:43:25.066380 14546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:43:25.069625 14546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:43:25.069859 14546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:43:25.078063 14546 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:38849
I20260925 22:43:25.078065 14613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:38849 every 8 connection(s)
I20260925 22:43:25.079268 14546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:43:25.080704 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 14546
I20260925 22:43:25.080843 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:43:25.081152 2347 external_mini_cluster.cc:1428] Setting key 946a3ac00375ba9ea63b01d3390750c4
I20260925 22:43:25.082640 14614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:43:25.090310 14614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e: Bootstrap starting.
Sep 25 22:43:25 dist-test-slave-csnl krb5kdc[14102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:43:25.093410 14614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e: Neither blocks nor log segments found. Creating new log.
I20260925 22:43:25.094543 14614 log.cc:824] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e: Log is configured to *not* fsync() on all Append() calls
I20260925 22:43:25.097177 14614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e: No bootstrap required, opened a new log
I20260925 22:43:25.098752 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:25.082753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39200 (local address 127.2.74.254:38849)
0925 22:43:25.083309 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:25.083330 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:25.083375 (+ 45us) server_negotiation.cc:407] Connection header received
0925 22:43:25.084268 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:25.084292 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:25.084761 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:25.085240 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:25.086267 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:25.087798 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:25.088794 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:25.089541 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:25.089724 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:25.092745 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:25.092752 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:25.092771 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:25.092812 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:25.095396 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:25.095967 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:25.095973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:25.095981 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:25.096123 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:25.096507 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:25.096510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:25.096512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:25.097050 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:25.097246 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:25.097632 (+ 386us) server_negotiation.cc:299] Negotiation successful
0925 22:43:25.097959 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":193,"threads_started":1}
I20260925 22:43:25.100878 14614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110b9b188a944147953c1a74fe93cb9e" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38849 } }
I20260925 22:43:25.101233 14614 raft_consensus.cc:374] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:43:25.101349 14614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 110b9b188a944147953c1a74fe93cb9e, State: Initialized, Role: FOLLOWER
I20260925 22:43:25.101838 14614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110b9b188a944147953c1a74fe93cb9e" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38849 } }
I20260925 22:43:25.102011 14614 raft_consensus.cc:388] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:43:25.102102 14614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:43:25.102240 14614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:43:25.103390 14614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110b9b188a944147953c1a74fe93cb9e" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38849 } }
I20260925 22:43:25.103813 14614 leader_election.cc:302] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 110b9b188a944147953c1a74fe93cb9e; no voters:
I20260925 22:43:25.104233 14614 leader_election.cc:288] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:43:25.104357 14619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:43:25.104625 14619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [term 1 LEADER]: Becoming Leader. State: Replica: 110b9b188a944147953c1a74fe93cb9e, State: Running, Role: LEADER
I20260925 22:43:25.105062 14619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110b9b188a944147953c1a74fe93cb9e" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38849 } }
I20260925 22:43:25.105712 14614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:43:25.106932 14621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 110b9b188a944147953c1a74fe93cb9e. Latest consensus state: current_term: 1 leader_uuid: "110b9b188a944147953c1a74fe93cb9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110b9b188a944147953c1a74fe93cb9e" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38849 } } }
I20260925 22:43:25.107139 14621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [sys.catalog]: This master's current role is: LEADER
I20260925 22:43:25.107575 14628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:43:25.107532 14620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "110b9b188a944147953c1a74fe93cb9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110b9b188a944147953c1a74fe93cb9e" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38849 } } }
I20260925 22:43:25.107739 14620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [sys.catalog]: This master's current role is: LEADER
I20260925 22:43:25.108084 14614 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 22:43:25.114074 14628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:43:25.122627 14628 catalog_manager.cc:1347] Generated new cluster ID: 71d48a25e5784611855b39a66aa02f8f
I20260925 22:43:25.122722 14628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:43:25.140597 14628 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:43:25.142131 14628 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:43:25.157363 14628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e: Generated new TSK 0
I20260925 22:43:25.158479 14628 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
I20260925 22:43:25.879921 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.74.212:52161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:0
--local_ip_for_outbound_sockets=127.2.74.193
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:38849
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:43:26.028385 14659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:43:26.028805 14659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:43:26.028895 14659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:43:26.035427 14659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:43:26.035528 14659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:43:26.035625 14659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:43:26.044458 14659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.74.212:52161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:38849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:43:26.045938 14659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:43:26.047413 14659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:43:26.059355 14666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:26.059341 14667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:26.059341 14669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:26.059923 14659 server_base.cc:1034] running on GCE node
I20260925 22:43:26.060436 14659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:43:26.061182 14659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:43:26.062369 14659 hybrid_clock.cc:648] HybridClock initialized: now 1790376206062346 us; error 41 us; skew 500 ppm
Sep 25 22:43:26 dist-test-slave-csnl krb5kdc[14102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376206, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:43:26.066048 14659 init.cc:377] Logged in from keytab as kudu/127.2.74.193@KRBTEST.COM (short username kudu)
I20260925 22:43:26.067284 14659 webserver.cc:460] Webserver started at http://127.2.74.193:41147/ using document root <none> and password file <none>
I20260925 22:43:26.068009 14659 fs_manager.cc:359] Metadata directory not provided
I20260925 22:43:26.068104 14659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:43:26.068339 14659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:43:26 dist-test-slave-csnl krb5kdc[14102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376206, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for HTTP/127.2.74.212@KRBTEST.COM
I20260925 22:43:26.086800 14659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/data/instance:
uuid: "cc548125786d48718d46eaceb3f0a3bf"
format_stamp: "Formatted at 2026-09-25 22:43:26 on dist-test-slave-csnl"
server_key: "327248029b4f4d34f0e8c3c4bf1e18d9"
server_key_iv: "f17c8841eeb23368c3c34aeb624c1c2a"
server_key_version: "kuduclusterkey@0"
I20260925 22:43:26.089659 14659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance:
uuid: "cc548125786d48718d46eaceb3f0a3bf"
format_stamp: "Formatted at 2026-09-25 22:43:26 on dist-test-slave-csnl"
server_key: "327248029b4f4d34f0e8c3c4bf1e18d9"
server_key_iv: "f17c8841eeb23368c3c34aeb624c1c2a"
server_key_version: "kuduclusterkey@0"
I20260925 22:43:26.101006 14659 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.004s sys 0.000s
I20260925 22:43:26.123070 14676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:26.126766 14659 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.000s
I20260925 22:43:26.126977 14659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "cc548125786d48718d46eaceb3f0a3bf"
format_stamp: "Formatted at 2026-09-25 22:43:26 on dist-test-slave-csnl"
server_key: "327248029b4f4d34f0e8c3c4bf1e18d9"
server_key_iv: "f17c8841eeb23368c3c34aeb624c1c2a"
server_key_version: "kuduclusterkey@0"
I20260925 22:43:26.127143 14659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:43:26.151027 14659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:43:26.155009 14659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:43:26.155215 14659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:43:26.155953 14659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:43:26.157117 14659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:43:26.157189 14659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:26.157260 14659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:43:26.157287 14659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:26.169840 14659 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:44019
I20260925 22:43:26.172196 14659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:43:26.173532 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 14659
I20260925 22:43:26.173679 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance
I20260925 22:43:26.173830 14791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:44019 every 8 connection(s)
I20260925 22:43:26.173985 2347 external_mini_cluster.cc:1428] Setting key 18586228b165671edac2e9ee953432f3
WARNING: no policy specified for kudu/127.2.74.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.194@KRBTEST.COM" created.
Sep 25 22:43:26 dist-test-slave-csnl krb5kdc[14102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376206, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:43:26.199940 14795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:26.180175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:34883 (local address 127.2.74.254:38849)
0925 22:43:26.182858 (+ 2683us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:26.182863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:26.183464 (+ 601us) server_negotiation.cc:407] Connection header received
0925 22:43:26.184851 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:26.184856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:26.184921 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:26.184985 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:26.186824 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:26.187674 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:26.189498 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:26.189879 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:26.190003 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:26.194854 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:26.194860 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:26.194862 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:26.194897 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:26.197001 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:26.197749 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:26.197754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:26.197756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:26.197860 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:26.198351 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:26.198355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:26.198357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:26.198630 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:26.198713 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:26.199580 (+ 867us) server_negotiation.cc:299] Negotiation successful
0925 22:43:26.199741 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2577,"thread_start_us":93,"threads_started":1}
I20260925 22:43:26.202272 14796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:26.180757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:38849 (local address 127.2.74.193:34883)
0925 22:43:26.183249 (+ 2492us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:26.183324 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:26.184548 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:26.185502 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:26.185512 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:26.185877 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:43:26.186619 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:26.186645 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:26.187841 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:26.187845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:26.189336 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:26.189348 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:26.190075 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:26.190079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:26.190292 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:26.192640 (+ 2348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:43:26.192667 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:43:26.194588 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:43:26.197165 (+ 2577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:26.197174 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:26.197196 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:26.197613 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:26.197991 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:26.197996 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:26.197998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:26.198217 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:26.198812 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:43:26.198819 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:43:26.199249 (+ 430us) client_negotiation.cc:769] Sending connection context
0925 22:43:26.200561 (+ 1312us) client_negotiation.cc:240] Negotiation successful
0925 22:43:26.200987 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2210,"thread_start_us":149,"threads_started":1}
I20260925 22:43:26.204092 14792 heartbeater.cc:347] Connected to a master server at 127.2.74.254:38849
I20260925 22:43:26.204463 14792 heartbeater.cc:464] Registering TS with master...
I20260925 22:43:26.205121 14792 heartbeater.cc:511] Master 127.2.74.254:38849 requested a full tablet report, sending...
I20260925 22:43:26.207190 14578 ts_manager.cc:194] Registered new tserver with Master: cc548125786d48718d46eaceb3f0a3bf (127.2.74.193:44019)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
I20260925 22:43:26.208783 14578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.193@KRBTEST.COM'} at 127.2.74.193:34883
I20260925 22:43:26.209560 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.74.212:52161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:0
--local_ip_for_outbound_sockets=127.2.74.194
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:38849
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:43:26 dist-test-slave-csnl krb5kdc[14102](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790376206, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 22:43:26.367039 14798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:43:26.367369 14798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:43:26.367466 14798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:43:26.372961 14798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:43:26.373067 14798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:43:26.373181 14798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:43:26.379642 14798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.74.212:52161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:38849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:43:26.380937 14798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:43:26.382046 14798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:43:26.392496 14804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:26.392496 14807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:26.392961 14805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:26.393733 14798 server_base.cc:1034] running on GCE node
I20260925 22:43:26.394295 14798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:43:26.395198 14798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:43:26.396416 14798 hybrid_clock.cc:648] HybridClock initialized: now 1790376206396381 us; error 62 us; skew 500 ppm
Sep 25 22:43:26 dist-test-slave-csnl krb5kdc[14102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376206, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:43:26.400444 14798 init.cc:377] Logged in from keytab as kudu/127.2.74.194@KRBTEST.COM (short username kudu)
I20260925 22:43:26.401713 14798 webserver.cc:460] Webserver started at http://127.2.74.194:40795/ using document root <none> and password file <none>
I20260925 22:43:26.402452 14798 fs_manager.cc:359] Metadata directory not provided
I20260925 22:43:26.402530 14798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:43:26.402769 14798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:43:26 dist-test-slave-csnl krb5kdc[14102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376206, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for HTTP/127.2.74.212@KRBTEST.COM
I20260925 22:43:26.420076 14798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/data/instance:
uuid: "10883a25fd55430d9ced01c7987c7571"
format_stamp: "Formatted at 2026-09-25 22:43:26 on dist-test-slave-csnl"
server_key: "bd29e103f4a51d6231518f71dace9a9c"
server_key_iv: "1e4305385da54fc87e6fd5339a20c218"
server_key_version: "kuduclusterkey@0"
I20260925 22:43:26.420890 14798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance:
uuid: "10883a25fd55430d9ced01c7987c7571"
format_stamp: "Formatted at 2026-09-25 22:43:26 on dist-test-slave-csnl"
server_key: "bd29e103f4a51d6231518f71dace9a9c"
server_key_iv: "1e4305385da54fc87e6fd5339a20c218"
server_key_version: "kuduclusterkey@0"
I20260925 22:43:26.426659 14798 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 22:43:26.455502 14814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:26.457448 14798 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260925 22:43:26.457609 14798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "10883a25fd55430d9ced01c7987c7571"
format_stamp: "Formatted at 2026-09-25 22:43:26 on dist-test-slave-csnl"
server_key: "bd29e103f4a51d6231518f71dace9a9c"
server_key_iv: "1e4305385da54fc87e6fd5339a20c218"
server_key_version: "kuduclusterkey@0"
I20260925 22:43:26.457777 14798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:43:26.490911 14798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:43:26.494922 14798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:43:26.495182 14798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:43:26.495957 14798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:43:26.497078 14798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:43:26.497150 14798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:26.497217 14798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:43:26.497251 14798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:26.508051 14798 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:42495
I20260925 22:43:26.509290 14798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:43:26.516927 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 14798
I20260925 22:43:26.517079 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance
I20260925 22:43:26.517401 2347 external_mini_cluster.cc:1428] Setting key 9703cb29de8f37481b7ba55bf0e4b0b6
I20260925 22:43:26.526298 14927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:42495 every 8 connection(s)
WARNING: no policy specified for kudu/127.2.74.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.195@KRBTEST.COM" created.
Sep 25 22:43:26 dist-test-slave-csnl krb5kdc[14102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376206, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:43:26.547948 14795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:26.530036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:46821 (local address 127.2.74.254:38849)
0925 22:43:26.530170 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:26.530176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:26.530922 (+ 746us) server_negotiation.cc:407] Connection header received
0925 22:43:26.532325 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:26.532329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:26.532392 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:26.532444 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:26.534119 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:26.534941 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:26.536668 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:26.537030 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:26.537140 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:26.542321 (+ 5181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:26.542326 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:26.542329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:26.542364 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:26.544363 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:26.545302 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:26.545307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:26.545309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:26.545416 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:26.546013 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:26.546018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:26.546020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:26.546324 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:26.546424 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:26.547480 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0925 22:43:26.547703 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 22:43:26.550522 14931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:26.528848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:38849 (local address 127.2.74.194:46821)
0925 22:43:26.530750 (+ 1902us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:26.530803 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:26.532044 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:26.532876 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:26.532885 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:26.533243 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:43:26.533935 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:26.533953 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:26.535117 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:26.535121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:26.536516 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:26.536530 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:26.539137 (+ 2607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:26.539143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:26.539386 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:26.540196 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:43:26.540220 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:43:26.542074 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:43:26.544579 (+ 2505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:26.544590 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:26.544619 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:26.545139 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:26.545553 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:26.545558 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:26.545562 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:26.545853 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:26.546540 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:43:26.546551 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:43:26.547085 (+ 534us) client_negotiation.cc:769] Sending connection context
0925 22:43:26.548507 (+ 1422us) client_negotiation.cc:240] Negotiation successful
0925 22:43:26.549049 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1665,"thread_start_us":175,"threads_started":1}
I20260925 22:43:26.552388 14928 heartbeater.cc:347] Connected to a master server at 127.2.74.254:38849
I20260925 22:43:26.552780 14928 heartbeater.cc:464] Registering TS with master...
I20260925 22:43:26.553471 14928 heartbeater.cc:511] Master 127.2.74.254:38849 requested a full tablet report, sending...
I20260925 22:43:26.559162 14578 ts_manager.cc:194] Registered new tserver with Master: 10883a25fd55430d9ced01c7987c7571 (127.2.74.194:42495)
I20260925 22:43:26.560006 14578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.194@KRBTEST.COM'} at 127.2.74.194:46821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
I20260925 22:43:26.576364 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.74.212:52161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:0
--local_ip_for_outbound_sockets=127.2.74.195
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:38849
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:43:26 dist-test-slave-csnl krb5kdc[14102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376206, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for HTTP/127.2.74.212@KRBTEST.COM
W20260925 22:43:26.751930 14933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:43:26.752336 14933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:43:26.752398 14933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:43:26.758600 14933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:43:26.758702 14933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:43:26.758791 14933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:43:26.767117 14933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.74.212:52161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:38849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:43:26.768579 14933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:43:26.769955 14933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:43:26.783785 14939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:26.784224 14933 server_base.cc:1034] running on GCE node
W20260925 22:43:26.784883 14941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:26.787147 14938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:26.787837 14933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:43:26.788712 14933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:43:26.796156 14933 hybrid_clock.cc:648] HybridClock initialized: now 1790376206796053 us; error 103 us; skew 500 ppm
Sep 25 22:43:26 dist-test-slave-csnl krb5kdc[14102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376206, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:43:26.800420 14933 init.cc:377] Logged in from keytab as kudu/127.2.74.195@KRBTEST.COM (short username kudu)
I20260925 22:43:26.801641 14933 webserver.cc:460] Webserver started at http://127.2.74.195:45071/ using document root <none> and password file <none>
I20260925 22:43:26.802434 14933 fs_manager.cc:359] Metadata directory not provided
I20260925 22:43:26.802572 14933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:43:26.802858 14933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:43:26 dist-test-slave-csnl krb5kdc[14102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376206, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for HTTP/127.2.74.212@KRBTEST.COM
I20260925 22:43:26.826617 14933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/data/instance:
uuid: "9ec178f52e35455381d6e746b7675da3"
format_stamp: "Formatted at 2026-09-25 22:43:26 on dist-test-slave-csnl"
server_key: "576b967ea8e590c414814ebc3eb88119"
server_key_iv: "0fb7b61acff26b2d4a9de47772082b6d"
server_key_version: "kuduclusterkey@0"
I20260925 22:43:26.827445 14933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance:
uuid: "9ec178f52e35455381d6e746b7675da3"
format_stamp: "Formatted at 2026-09-25 22:43:26 on dist-test-slave-csnl"
server_key: "576b967ea8e590c414814ebc3eb88119"
server_key_iv: "0fb7b61acff26b2d4a9de47772082b6d"
server_key_version: "kuduclusterkey@0"
I20260925 22:43:26.839148 14933 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.005s sys 0.000s
I20260925 22:43:26.863293 14948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:26.865355 14933 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.004s sys 0.000s
I20260925 22:43:26.865526 14933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "9ec178f52e35455381d6e746b7675da3"
format_stamp: "Formatted at 2026-09-25 22:43:26 on dist-test-slave-csnl"
server_key: "576b967ea8e590c414814ebc3eb88119"
server_key_iv: "0fb7b61acff26b2d4a9de47772082b6d"
server_key_version: "kuduclusterkey@0"
I20260925 22:43:26.865680 14933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:43:26.894079 14933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:43:26.897949 14933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:43:26.898149 14933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:43:26.898860 14933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:43:26.899904 14933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:43:26.899963 14933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:26.900017 14933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:43:26.900044 14933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:26.910471 14933 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:36405
I20260925 22:43:26.910527 15061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:36405 every 8 connection(s)
I20260925 22:43:26.911725 14933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:43:26.912123 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 14933
I20260925 22:43:26.912252 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance
I20260925 22:43:26.912565 2347 external_mini_cluster.cc:1428] Setting key 7d41bc5482cfbaee3eab64961492ab33
Sep 25 22:43:26 dist-test-slave-csnl krb5kdc[14102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376206, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:43:26.931406 14795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:26.915656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:51445 (local address 127.2.74.254:38849)
0925 22:43:26.915805 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:26.915810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:26.916895 (+ 1085us) server_negotiation.cc:407] Connection header received
0925 22:43:26.918104 (+ 1209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:26.918107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:26.918196 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:26.918297 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:26.919938 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:26.920838 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:26.922664 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:26.923065 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:26.923212 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:26.926516 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:26.926520 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:26.926523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:26.926555 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:26.928567 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:26.929297 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:26.929301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:26.929303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:26.929411 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:26.929936 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:26.929940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:26.929942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:26.930191 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:26.930298 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:26.931082 (+ 784us) server_negotiation.cc:299] Negotiation successful
0925 22:43:26.931230 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:43:26.932942 15064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:26.916126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:38849 (local address 127.2.74.195:51445)
0925 22:43:26.916681 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:26.916764 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:26.917821 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:26.918509 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:26.918519 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:26.918905 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:43:26.919746 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:26.919768 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:26.920993 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:26.920997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:26.922487 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:26.922500 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:26.923261 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:26.923266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:26.923480 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:26.924258 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:43:26.924280 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:43:26.926280 (+ 2000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:43:26.928721 (+ 2441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:26.928729 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:26.928750 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:26.929158 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:26.929543 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:26.929546 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:26.929548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:26.929808 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:26.930328 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:43:26.930336 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:43:26.930734 (+ 398us) client_negotiation.cc:769] Sending connection context
0925 22:43:26.931020 (+ 286us) client_negotiation.cc:240] Negotiation successful
0925 22:43:26.931404 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":138,"threads_started":1}
I20260925 22:43:26.934497 15062 heartbeater.cc:347] Connected to a master server at 127.2.74.254:38849
I20260925 22:43:26.934798 15062 heartbeater.cc:464] Registering TS with master...
I20260925 22:43:26.935454 15062 heartbeater.cc:511] Master 127.2.74.254:38849 requested a full tablet report, sending...
I20260925 22:43:26.936821 14578 ts_manager.cc:194] Registered new tserver with Master: 9ec178f52e35455381d6e746b7675da3 (127.2.74.195:36405)
I20260925 22:43:26.937649 14578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.195@KRBTEST.COM'} at 127.2.74.195:51445
I20260925 22:43:26.940398 2347 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:43:26.941284 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 14659
I20260925 22:43:26.947352 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 14798
I20260925 22:43:26.953370 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 14933
I20260925 22:43:26.960423 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 14546
I20260925 22:43:26.967458 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:38849
--webserver_interface=127.2.74.254
--webserver_port=35263
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:38849
--ranger_config_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.74.212:52161/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.74.212:52161/kms with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:43:27.084636 15065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:43:27.084937 15065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:43:27.084978 15065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:43:27.089318 15065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:43:27.089406 15065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:43:27.089432 15065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:43:27.089448 15065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:43:27.089464 15065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:43:27.095614 15065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.74.212:52161/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:38849
--ranger_config_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:38849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=35263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:43:27.096840 15065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:43:27.097991 15065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:43:27.107488 15073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:27.107527 15070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:27.107682 15065 server_base.cc:1034] running on GCE node
W20260925 22:43:27.107492 15071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:27.108510 15065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:43:27.109831 15065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:43:27.111044 15065 hybrid_clock.cc:648] HybridClock initialized: now 1790376207111005 us; error 59 us; skew 500 ppm
Sep 25 22:43:27 dist-test-slave-csnl krb5kdc[14102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376207, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:43:27.114044 15065 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:43:27.115231 15065 webserver.cc:460] Webserver started at http://127.2.74.254:35263/ using document root <none> and password file <none>
I20260925 22:43:27.115867 15065 fs_manager.cc:359] Metadata directory not provided
I20260925 22:43:27.115984 15065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 22:43:27 dist-test-slave-csnl krb5kdc[14102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376207, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for HTTP/127.2.74.212@KRBTEST.COM
I20260925 22:43:27.184209 15065 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.002s
I20260925 22:43:27.187070 15081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:27.188575 15065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:43:27.188748 15065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "110b9b188a944147953c1a74fe93cb9e"
format_stamp: "Formatted at 2026-09-25 22:43:25 on dist-test-slave-csnl"
server_key: "be4010ea295f90b48c112bf9132d7aee"
server_key_iv: "ff0553a1248b59d2ffd186de4f413aa2"
server_key_version: "kuduclusterkey@0"
I20260925 22:43:27.189364 15065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:43:27.209743 15065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:43:27.212575 15065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:43:27.212790 15065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:43:27.220392 15065 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:38849
I20260925 22:43:27.220414 15133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:38849 every 8 connection(s)
I20260925 22:43:27.221552 15065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:43:27.225912 15134 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:43:27.226559 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 15065
I20260925 22:43:27.227105 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.74.212:52161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:44019
--local_ip_for_outbound_sockets=127.2.74.193
--tserver_master_addrs=127.2.74.254:38849
--webserver_port=41147
--webserver_interface=127.2.74.193
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.74.212:52161/kms with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:43:27.228413 15134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e: Bootstrap starting.
I20260925 22:43:27.245682 15134 log.cc:824] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e: Log is configured to *not* fsync() on all Append() calls
I20260925 22:43:27.252846 15134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:43:27.253314 15134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e: Bootstrap complete.
I20260925 22:43:27.257786 15134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110b9b188a944147953c1a74fe93cb9e" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38849 } }
I20260925 22:43:27.258216 15134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 110b9b188a944147953c1a74fe93cb9e, State: Initialized, Role: FOLLOWER
I20260925 22:43:27.258767 15134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110b9b188a944147953c1a74fe93cb9e" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38849 } }
I20260925 22:43:27.258898 15134 raft_consensus.cc:388] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:43:27.259012 15134 raft_consensus.cc:482] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:43:27.259114 15134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:43:27.261286 15134 raft_consensus.cc:504] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110b9b188a944147953c1a74fe93cb9e" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38849 } }
I20260925 22:43:27.261693 15134 leader_election.cc:302] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 110b9b188a944147953c1a74fe93cb9e; no voters:
I20260925 22:43:27.262041 15134 leader_election.cc:288] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 22:43:27.262276 15138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [term 2 FOLLOWER]: Leader election won for term 2
I20260925 22:43:27.262614 15138 raft_consensus.cc:686] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [term 2 LEADER]: Becoming Leader. State: Replica: 110b9b188a944147953c1a74fe93cb9e, State: Running, Role: LEADER
I20260925 22:43:27.263084 15138 consensus_queue.cc:237] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110b9b188a944147953c1a74fe93cb9e" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38849 } }
I20260925 22:43:27.263463 15134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:43:27.264770 15139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "110b9b188a944147953c1a74fe93cb9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110b9b188a944147953c1a74fe93cb9e" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38849 } } }
I20260925 22:43:27.264816 15140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 110b9b188a944147953c1a74fe93cb9e. Latest consensus state: current_term: 2 leader_uuid: "110b9b188a944147953c1a74fe93cb9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110b9b188a944147953c1a74fe93cb9e" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38849 } } }
I20260925 22:43:27.264901 15139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [sys.catalog]: This master's current role is: LEADER
I20260925 22:43:27.264917 15140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e [sys.catalog]: This master's current role is: LEADER
I20260925 22:43:27.265235 15147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:43:27.268586 15147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:43:27.269124 15134 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 22:43:27.269254 15147 catalog_manager.cc:1259] Loaded cluster ID: 71d48a25e5784611855b39a66aa02f8f
I20260925 22:43:27.269318 15147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:43:27.271772 15147 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:43:27.272490 15147 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e: Loaded TSK: 0
I20260925 22:43:27.273793 15147 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:43:27.382562 15136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:43:27.383005 15136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:43:27.383081 15136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:43:27.389916 15136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:43:27.390028 15136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:43:27.390127 15136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:43:27.399322 15136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.74.212:52161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:44019
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=41147
--tserver_master_addrs=127.2.74.254:38849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:43:27.400887 15136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:43:27.402415 15136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:43:27.416669 15172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:27.417248 15171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:27.417399 15174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:27.418833 15136 server_base.cc:1034] running on GCE node
I20260925 22:43:27.419435 15136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:43:27.420367 15136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:43:27.421615 15136 hybrid_clock.cc:648] HybridClock initialized: now 1790376207421539 us; error 103 us; skew 500 ppm
Sep 25 22:43:27 dist-test-slave-csnl krb5kdc[14102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376207, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:43:27.425402 15136 init.cc:377] Logged in from keytab as kudu/127.2.74.193@KRBTEST.COM (short username kudu)
I20260925 22:43:27.426782 15136 webserver.cc:460] Webserver started at http://127.2.74.193:41147/ using document root <none> and password file <none>
I20260925 22:43:27.427654 15136 fs_manager.cc:359] Metadata directory not provided
I20260925 22:43:27.427776 15136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 22:43:27 dist-test-slave-csnl krb5kdc[14102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376207, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for HTTP/127.2.74.212@KRBTEST.COM
I20260925 22:43:27.451719 15136 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:43:27.454711 15182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:27.456243 15136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:43:27.456357 15136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "cc548125786d48718d46eaceb3f0a3bf"
format_stamp: "Formatted at 2026-09-25 22:43:26 on dist-test-slave-csnl"
server_key: "327248029b4f4d34f0e8c3c4bf1e18d9"
server_key_iv: "f17c8841eeb23368c3c34aeb624c1c2a"
server_key_version: "kuduclusterkey@0"
I20260925 22:43:27.456990 15136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:43:27.471700 15136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:43:27.475342 15136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:43:27.475561 15136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:43:27.476336 15136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:43:27.477524 15136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:43:27.477599 15136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:27.477662 15136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:43:27.477694 15136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:27.489591 15136 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:44019
I20260925 22:43:27.489776 15295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:44019 every 8 connection(s)
I20260925 22:43:27.490880 15136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:43:27.491283 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 15136
I20260925 22:43:27.491822 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.74.212:52161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:42495
--local_ip_for_outbound_sockets=127.2.74.194
--tserver_master_addrs=127.2.74.254:38849
--webserver_port=40795
--webserver_interface=127.2.74.194
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.74.212:52161/kms with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:43:27 dist-test-slave-csnl krb5kdc[14102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376207, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:43:27.512794 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:27.493818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:38849 (local address 127.2.74.193:43539)
0925 22:43:27.494443 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:27.494501 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:27.496253 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:27.497547 (+ 1294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:27.497557 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:27.497942 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:43:27.498598 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:27.498629 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:27.500365 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:27.500371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:27.501459 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:27.501476 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:27.503075 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:27.503083 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:27.503353 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:27.504192 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:43:27.504217 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:43:27.505935 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:43:27.508981 (+ 3046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:27.508989 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:27.509008 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:27.509383 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:27.509844 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:27.509848 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:27.509850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:27.510012 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:27.510691 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:43:27.510698 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:43:27.511049 (+ 351us) client_negotiation.cc:769] Sending connection context
0925 22:43:27.511318 (+ 269us) client_negotiation.cc:240] Negotiation successful
0925 22:43:27.511708 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":160,"threads_started":1}
I20260925 22:43:27.512777 15300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:27.495327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:43539 (local address 127.2.74.254:38849)
0925 22:43:27.495788 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:27.495809 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:27.495861 (+ 52us) server_negotiation.cc:407] Connection header received
0925 22:43:27.496742 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:27.496753 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:27.497097 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:27.497461 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:27.498886 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:27.500183 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:27.501657 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:27.502541 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:27.502746 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:27.506262 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:27.506271 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:27.506292 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:27.506334 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:27.508784 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:27.509571 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:27.509582 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:27.509587 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:27.509723 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:27.510124 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:27.510128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:27.510130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:27.510561 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:27.510721 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:27.511453 (+ 732us) server_negotiation.cc:299] Negotiation successful
0925 22:43:27.511722 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":134,"threads_started":1}
I20260925 22:43:27.516263 15296 heartbeater.cc:347] Connected to a master server at 127.2.74.254:38849
I20260925 22:43:27.516562 15296 heartbeater.cc:464] Registering TS with master...
I20260925 22:43:27.517237 15296 heartbeater.cc:511] Master 127.2.74.254:38849 requested a full tablet report, sending...
W20260925 22:43:27.519902 15296 heartbeater.cc:644] Failed to heartbeat to 127.2.74.254:38849 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 22:43:27.520061 15296 heartbeater.cc:464] Registering TS with master...
I20260925 22:43:27.520290 15296 heartbeater.cc:511] Master 127.2.74.254:38849 requested a full tablet report, sending...
I20260925 22:43:27.521274 15296 heartbeater.cc:464] Registering TS with master...
I20260925 22:43:27.521478 15296 heartbeater.cc:511] Master 127.2.74.254:38849 requested a full tablet report, sending...
W20260925 22:43:27.522337 15296 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:43:27.672042 15298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:43:27.672461 15298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:43:27.672533 15298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:43:27.679219 15298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:43:27.679325 15298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:43:27.679447 15298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:43:27.688128 15298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.74.212:52161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:42495
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=40795
--tserver_master_addrs=127.2.74.254:38849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:43:27.689690 15298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:43:27.691202 15298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:43:27.704495 15305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:27.704491 15308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:27.704483 15306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:27.705192 15298 server_base.cc:1034] running on GCE node
I20260925 22:43:27.705655 15298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:43:27.706437 15298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:43:27.707617 15298 hybrid_clock.cc:648] HybridClock initialized: now 1790376207707590 us; error 47 us; skew 500 ppm
Sep 25 22:43:27 dist-test-slave-csnl krb5kdc[14102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376207, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:43:27.710757 15298 init.cc:377] Logged in from keytab as kudu/127.2.74.194@KRBTEST.COM (short username kudu)
I20260925 22:43:27.711818 15298 webserver.cc:460] Webserver started at http://127.2.74.194:40795/ using document root <none> and password file <none>
I20260925 22:43:27.712416 15298 fs_manager.cc:359] Metadata directory not provided
I20260925 22:43:27.712493 15298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 22:43:27 dist-test-slave-csnl krb5kdc[14102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376207, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for HTTP/127.2.74.212@KRBTEST.COM
I20260925 22:43:27.746847 15298 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.003s sys 0.000s
I20260925 22:43:27.750644 15316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:27.752679 15298 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:43:27.752854 15298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "10883a25fd55430d9ced01c7987c7571"
format_stamp: "Formatted at 2026-09-25 22:43:26 on dist-test-slave-csnl"
server_key: "bd29e103f4a51d6231518f71dace9a9c"
server_key_iv: "1e4305385da54fc87e6fd5339a20c218"
server_key_version: "kuduclusterkey@0"
I20260925 22:43:27.753683 15298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:43:27.771412 15298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:43:27.775485 15298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:43:27.775722 15298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:43:27.776541 15298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:43:27.777693 15298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:43:27.777770 15298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:27.777838 15298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:43:27.777874 15298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:27.795691 15298 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:42495
I20260925 22:43:27.797298 15298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:43:27.799083 15429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:42495 every 8 connection(s)
I20260925 22:43:27.802438 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 15298
I20260925 22:43:27.803092 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.74.212:52161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:36405
--local_ip_for_outbound_sockets=127.2.74.195
--tserver_master_addrs=127.2.74.254:38849
--webserver_port=45071
--webserver_interface=127.2.74.195
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.74.212:52161/kms with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:43:27 dist-test-slave-csnl krb5kdc[14102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376207, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:43:27.818907 15300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:27.803372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:54011 (local address 127.2.74.254:38849)
0925 22:43:27.803564 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:27.803571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:27.804041 (+ 470us) server_negotiation.cc:407] Connection header received
0925 22:43:27.805419 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:27.805424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:27.805488 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:27.805559 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:27.807477 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:27.808335 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:27.810225 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:27.810592 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:27.810685 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:27.814056 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:27.814061 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:27.814064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:27.814097 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:27.815995 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:27.816810 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:27.816815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:27.816816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:27.816933 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:27.817427 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:27.817430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:27.817432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:27.817686 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:27.817773 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:27.818539 (+ 766us) server_negotiation.cc:299] Negotiation successful
0925 22:43:27.818711 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:43:27.820744 15432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:27.802067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:38849 (local address 127.2.74.194:54011)
0925 22:43:27.803835 (+ 1768us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:27.803915 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:27.805112 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:27.806051 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:27.806062 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:27.806458 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:43:27.807261 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:27.807285 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:27.808523 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:27.808526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:27.810062 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:27.810075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:27.810936 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:27.810943 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:27.811176 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:27.811941 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:43:27.811964 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:43:27.813795 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:43:27.816166 (+ 2371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:27.816175 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:27.816196 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:27.816649 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:27.817062 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:27.817065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:27.817067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:27.817297 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:27.817843 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:43:27.817851 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:43:27.818224 (+ 373us) client_negotiation.cc:769] Sending connection context
0925 22:43:27.819466 (+ 1242us) client_negotiation.cc:240] Negotiation successful
0925 22:43:27.819900 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1519,"thread_start_us":185,"threads_started":1}
I20260925 22:43:27.822149 15430 heartbeater.cc:347] Connected to a master server at 127.2.74.254:38849
I20260925 22:43:27.822420 15430 heartbeater.cc:464] Registering TS with master...
I20260925 22:43:27.823045 15430 heartbeater.cc:511] Master 127.2.74.254:38849 requested a full tablet report, sending...
W20260925 22:43:27.824980 15430 heartbeater.cc:644] Failed to heartbeat to 127.2.74.254:38849 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 22:43:27.825130 15430 heartbeater.cc:464] Registering TS with master...
I20260925 22:43:27.825340 15430 heartbeater.cc:511] Master 127.2.74.254:38849 requested a full tablet report, sending...
I20260925 22:43:27.826376 15430 heartbeater.cc:464] Registering TS with master...
I20260925 22:43:27.826603 15430 heartbeater.cc:511] Master 127.2.74.254:38849 requested a full tablet report, sending...
W20260925 22:43:27.827535 15430 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:43:27.988607 15433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:43:27.989071 15433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:43:27.989146 15433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:43:27.995776 15433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:43:27.995878 15433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:43:27.995976 15433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:43:28.004798 15433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.74.212:52161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:36405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=45071
--tserver_master_addrs=127.2.74.254:38849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:43:28.006394 15433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:43:28.007872 15433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:43:28.022215 15442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:28.022512 15439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:28.022697 15440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:28.023737 15433 server_base.cc:1034] running on GCE node
I20260925 22:43:28.024336 15433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:43:28.025266 15433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:43:28.026495 15433 hybrid_clock.cc:648] HybridClock initialized: now 1790376208026448 us; error 73 us; skew 500 ppm
Sep 25 22:43:28 dist-test-slave-csnl krb5kdc[14102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376208, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:43:28.030741 15433 init.cc:377] Logged in from keytab as kudu/127.2.74.195@KRBTEST.COM (short username kudu)
I20260925 22:43:28.032168 15433 webserver.cc:460] Webserver started at http://127.2.74.195:45071/ using document root <none> and password file <none>
I20260925 22:43:28.032992 15433 fs_manager.cc:359] Metadata directory not provided
I20260925 22:43:28.033125 15433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 22:43:28 dist-test-slave-csnl krb5kdc[14102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376208, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for HTTP/127.2.74.212@KRBTEST.COM
I20260925 22:43:28.052821 15433 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260925 22:43:28.059347 15450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:28.061425 15433 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.000s
I20260925 22:43:28.061607 15433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "9ec178f52e35455381d6e746b7675da3"
format_stamp: "Formatted at 2026-09-25 22:43:26 on dist-test-slave-csnl"
server_key: "576b967ea8e590c414814ebc3eb88119"
server_key_iv: "0fb7b61acff26b2d4a9de47772082b6d"
server_key_version: "kuduclusterkey@0"
I20260925 22:43:28.062446 15433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:43:28.081692 15433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:43:28.085831 15433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:43:28.086081 15433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:43:28.086886 15433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:43:28.088090 15433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:43:28.088162 15433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:28.088225 15433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:43:28.088258 15433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:28.100626 15433 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:36405
I20260925 22:43:28.100644 15563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:36405 every 8 connection(s)
I20260925 22:43:28.101725 15433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:43:28.105055 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 15433
Sep 25 22:43:28 dist-test-slave-csnl krb5kdc[14102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376208, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:43:28.119228 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:28.105775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54986 (local address 127.2.74.254:38849)
0925 22:43:28.106845 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:28.106856 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:28.107158 (+ 302us) server_negotiation.cc:407] Connection header received
0925 22:43:28.107342 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:28.107347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:28.107414 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:28.107480 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:28.108528 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:28.109994 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:28.111230 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:28.111534 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:28.111607 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:28.114257 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:28.114264 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:28.114270 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:28.114304 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:28.116290 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:28.117203 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:28.117206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:28.117208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:28.117300 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:28.117952 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:28.117955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:28.117956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:28.118218 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:28.118460 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:28.118874 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 22:43:28.119048 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":999,"thread_start_us":88,"threads_started":1}
I20260925 22:43:28.121152 15300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:28.103881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:36849 (local address 127.2.74.254:38849)
0925 22:43:28.104021 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:28.104026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:28.105015 (+ 989us) server_negotiation.cc:407] Connection header received
0925 22:43:28.106639 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:28.106643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:28.106705 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:28.106784 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:28.108528 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:28.109130 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:28.110767 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:28.111147 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:28.111227 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:28.114368 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:28.114374 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:28.114379 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:28.114428 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:28.117074 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:28.117700 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:28.117705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:28.117706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:28.117793 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:28.119799 (+ 2006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:28.119803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:28.119804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:28.120013 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:28.120072 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:28.120896 (+ 824us) server_negotiation.cc:299] Negotiation successful
0925 22:43:28.121024 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":57}
I20260925 22:43:28.122557 15566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:28.104358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:38849 (local address 127.2.74.195:36849)
0925 22:43:28.104851 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:28.104907 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:28.106402 (+ 1495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:28.107164 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:28.107177 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:28.107572 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:43:28.108371 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:28.108394 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:28.109269 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:28.109272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:28.110629 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:28.110640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:28.111438 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:28.111442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:28.111659 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:28.112419 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:43:28.112470 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:43:28.114191 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:43:28.117195 (+ 3004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:28.117202 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:28.117226 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:28.117589 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:28.117903 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:28.117908 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:28.118491 (+ 583us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:28.118720 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:28.120179 (+ 1459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:43:28.120187 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:43:28.120598 (+ 411us) client_negotiation.cc:769] Sending connection context
0925 22:43:28.121391 (+ 793us) client_negotiation.cc:240] Negotiation successful
0925 22:43:28.121728 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":151,"threads_started":1}
I20260925 22:43:28.123867 15564 heartbeater.cc:347] Connected to a master server at 127.2.74.254:38849
I20260925 22:43:28.124125 15564 heartbeater.cc:464] Registering TS with master...
I20260925 22:43:28.124665 15564 heartbeater.cc:511] Master 127.2.74.254:38849 requested a full tablet report, sending...
W20260925 22:43:28.126271 15564 heartbeater.cc:644] Failed to heartbeat to 127.2.74.254:38849 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 22:43:28.126430 15564 heartbeater.cc:464] Registering TS with master...
I20260925 22:43:28.126631 15564 heartbeater.cc:511] Master 127.2.74.254:38849 requested a full tablet report, sending...
I20260925 22:43:28.127517 15564 heartbeater.cc:464] Registering TS with master...
I20260925 22:43:28.127715 15564 heartbeater.cc:511] Master 127.2.74.254:38849 requested a full tablet report, sending...
W20260925 22:43:28.128499 15564 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 22:43:28 dist-test-slave-csnl krb5kdc[14102](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790376208, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:43:28.522616 15296 heartbeater.cc:464] Registering TS with master...
I20260925 22:43:28.522975 15296 heartbeater.cc:511] Master 127.2.74.254:38849 requested a full tablet report, sending...
I20260925 22:43:28.524209 15098 ts_manager.cc:194] Registered new tserver with Master: cc548125786d48718d46eaceb3f0a3bf (127.2.74.193:44019)
I20260925 22:43:28.525674 15098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.193@KRBTEST.COM'} at 127.2.74.193:43539
I20260925 22:43:28.827790 15430 heartbeater.cc:464] Registering TS with master...
I20260925 22:43:28.828094 15430 heartbeater.cc:511] Master 127.2.74.254:38849 requested a full tablet report, sending...
I20260925 22:43:28.828859 15098 ts_manager.cc:194] Registered new tserver with Master: 10883a25fd55430d9ced01c7987c7571 (127.2.74.194:42495)
I20260925 22:43:28.829670 15098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.194@KRBTEST.COM'} at 127.2.74.194:54011
Sep 25 22:43:28 dist-test-slave-csnl krb5kdc[14102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376208, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for HTTP/127.2.74.212@KRBTEST.COM
I20260925 22:43:29.128775 15564 heartbeater.cc:464] Registering TS with master...
I20260925 22:43:29.129094 15564 heartbeater.cc:511] Master 127.2.74.254:38849 requested a full tablet report, sending...
I20260925 22:43:29.129917 15098 ts_manager.cc:194] Registered new tserver with Master: 9ec178f52e35455381d6e746b7675da3 (127.2.74.195:36405)
I20260925 22:43:29.130704 15098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.195@KRBTEST.COM'} at 127.2.74.195:36849
I20260925 22:43:29.140590 2347 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:43:29.153033 15586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:29.142297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54998 (local address 127.2.74.254:38849)
0925 22:43:29.142541 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:29.142546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:29.142728 (+ 182us) server_negotiation.cc:407] Connection header received
0925 22:43:29.142862 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:29.142865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:29.142918 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:29.143058 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:29.143940 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:29.144894 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:29.146101 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:29.146465 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:29.146581 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:29.148477 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:29.148482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:29.148485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:29.148516 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:29.150492 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:29.151073 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:29.151078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:29.151083 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:29.151188 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:29.151533 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:29.151537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:29.151539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:29.151783 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:29.151911 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:29.152688 (+ 777us) server_negotiation.cc:299] Negotiation successful
0925 22:43:29.152833 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":81,"threads_started":1}
I20260925 22:43:29.158936 15098 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54998:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:43:29.164578 15098 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:43:29.192287 15591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:29.182323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:36405 (local address 127.0.0.1:44554)
0925 22:43:29.184274 (+ 1951us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:29.184340 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:29.184482 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:29.185269 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:29.185276 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:29.185299 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:43:29.185446 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:29.185453 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:29.186977 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:29.186981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:29.188877 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:29.188891 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:29.191162 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:29.191166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:29.191334 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:29.191851 (+ 517us) client_negotiation.cc:769] Sending connection context
0925 22:43:29.191970 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 22:43:29.192120 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1805,"mutex_wait_us":469,"thread_start_us":110,"threads_started":1}
I20260925 22:43:29.193148 15595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:29.183327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44554 (local address 127.2.74.195:36405)
0925 22:43:29.183666 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:29.183672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:29.184583 (+ 911us) server_negotiation.cc:407] Connection header received
0925 22:43:29.184676 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:29.184683 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:29.184918 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:29.185090 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:43:29.185586 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:29.186792 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:29.189081 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:29.189993 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:29.191418 (+ 1425us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:29.191584 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:29.192865 (+ 1281us) server_negotiation.cc:299] Negotiation successful
0925 22:43:29.192999 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":67,"threads_started":1}
I20260925 22:43:29.193759 15594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:29.183185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35184 (local address 127.2.74.193:44019)
0925 22:43:29.183555 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:29.183563 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:29.184760 (+ 1197us) server_negotiation.cc:407] Connection header received
0925 22:43:29.184997 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:29.185003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:29.185210 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:29.185351 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:43:29.185895 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:29.187182 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:29.189357 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:29.190490 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:29.190601 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:29.190816 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:29.193479 (+ 2663us) server_negotiation.cc:299] Negotiation successful
0925 22:43:29.193600 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":123,"threads_started":1}
I20260925 22:43:29.194386 15592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:29.182250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:44019 (local address 127.0.0.1:35184)
0925 22:43:29.184664 (+ 2414us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:29.184676 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:29.184833 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:29.185583 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:29.185588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:29.185612 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:43:29.185746 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:29.185755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:29.187359 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:29.187364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:29.189159 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:29.189174 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:29.190976 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:29.190982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:29.191211 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:29.192723 (+ 1512us) client_negotiation.cc:769] Sending connection context
0925 22:43:29.194123 (+ 1400us) client_negotiation.cc:240] Negotiation successful
0925 22:43:29.194216 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1642,"mutex_wait_us":163,"thread_start_us":130,"threads_started":1}
I20260925 22:43:29.197546 15597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:29.184341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60468 (local address 127.2.74.194:42495)
0925 22:43:29.191612 (+ 7271us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:29.191620 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:29.192009 (+ 389us) server_negotiation.cc:407] Connection header received
0925 22:43:29.192098 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:29.192104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:29.192358 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:29.192528 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:43:29.192957 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:29.194210 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:29.195699 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:29.196478 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:29.196584 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:29.196723 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:29.197312 (+ 589us) server_negotiation.cc:299] Negotiation successful
0925 22:43:29.197415 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7143,"thread_start_us":66,"threads_started":1}
I20260925 22:43:29.197211 15596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:29.184187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:42495 (local address 127.0.0.1:60468)
0925 22:43:29.191773 (+ 7586us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:29.191790 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:29.191891 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:29.192700 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:29.192703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:29.192720 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:43:29.192824 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:29.192830 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:29.194399 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:29.194405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:29.195557 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:29.195568 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:29.196787 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:29.196791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:29.196892 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:29.196926 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 22:43:29.197001 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 22:43:29.197074 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7515,"thread_start_us":86,"threads_started":1}
I20260925 22:43:29.198007 15493 tablet_service.cc:1467] Processing CreateTablet for tablet 890487a4220f4963b045301d315db8f4 (DEFAULT_TABLE table=test-table [id=50b9150d871144cd95e67216b0020fbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:43:29.201655 15364 tablet_service.cc:1467] Processing CreateTablet for tablet 890487a4220f4963b045301d315db8f4 (DEFAULT_TABLE table=test-table [id=50b9150d871144cd95e67216b0020fbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:43:29.201655 15230 tablet_service.cc:1467] Processing CreateTablet for tablet 890487a4220f4963b045301d315db8f4 (DEFAULT_TABLE table=test-table [id=50b9150d871144cd95e67216b0020fbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:43:29.202972 15364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 890487a4220f4963b045301d315db8f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:43:29.202976 15230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 890487a4220f4963b045301d315db8f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:43:29.207330 15493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 890487a4220f4963b045301d315db8f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:43:29.219313 15599 tablet_bootstrap.cc:492] T 890487a4220f4963b045301d315db8f4 P 10883a25fd55430d9ced01c7987c7571: Bootstrap starting.
I20260925 22:43:29.224323 15599 tablet_bootstrap.cc:654] T 890487a4220f4963b045301d315db8f4 P 10883a25fd55430d9ced01c7987c7571: Neither blocks nor log segments found. Creating new log.
I20260925 22:43:29.225378 15599 log.cc:824] T 890487a4220f4963b045301d315db8f4 P 10883a25fd55430d9ced01c7987c7571: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:43:29 dist-test-slave-csnl krb5kdc[14102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376208, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for HTTP/127.2.74.212@KRBTEST.COM
I20260925 22:43:29.230844 15598 tablet_bootstrap.cc:492] T 890487a4220f4963b045301d315db8f4 P cc548125786d48718d46eaceb3f0a3bf: Bootstrap starting.
I20260925 22:43:29.235844 15599 tablet_bootstrap.cc:492] T 890487a4220f4963b045301d315db8f4 P 10883a25fd55430d9ced01c7987c7571: No bootstrap required, opened a new log
I20260925 22:43:29.236123 15599 ts_tablet_manager.cc:1397] T 890487a4220f4963b045301d315db8f4 P 10883a25fd55430d9ced01c7987c7571: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20260925 22:43:29.239109 15598 tablet_bootstrap.cc:654] T 890487a4220f4963b045301d315db8f4 P cc548125786d48718d46eaceb3f0a3bf: Neither blocks nor log segments found. Creating new log.
I20260925 22:43:29.239720 15599 raft_consensus.cc:348] T 890487a4220f4963b045301d315db8f4 P 10883a25fd55430d9ced01c7987c7571 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc548125786d48718d46eaceb3f0a3bf" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44019 } } peers { permanent_uuid: "9ec178f52e35455381d6e746b7675da3" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36405 } } peers { permanent_uuid: "10883a25fd55430d9ced01c7987c7571" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 42495 } }
I20260925 22:43:29.240145 15598 log.cc:824] T 890487a4220f4963b045301d315db8f4 P cc548125786d48718d46eaceb3f0a3bf: Log is configured to *not* fsync() on all Append() calls
I20260925 22:43:29.240201 15599 raft_consensus.cc:374] T 890487a4220f4963b045301d315db8f4 P 10883a25fd55430d9ced01c7987c7571 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:43:29.240352 15599 raft_consensus.cc:729] T 890487a4220f4963b045301d315db8f4 P 10883a25fd55430d9ced01c7987c7571 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10883a25fd55430d9ced01c7987c7571, State: Initialized, Role: FOLLOWER
I20260925 22:43:29.241050 15600 tablet_bootstrap.cc:492] T 890487a4220f4963b045301d315db8f4 P 9ec178f52e35455381d6e746b7675da3: Bootstrap starting.
I20260925 22:43:29.240998 15599 consensus_queue.cc:260] T 890487a4220f4963b045301d315db8f4 P 10883a25fd55430d9ced01c7987c7571 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc548125786d48718d46eaceb3f0a3bf" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44019 } } peers { permanent_uuid: "9ec178f52e35455381d6e746b7675da3" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36405 } } peers { permanent_uuid: "10883a25fd55430d9ced01c7987c7571" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 42495 } }
I20260925 22:43:29.242295 15599 ts_tablet_manager.cc:1428] T 890487a4220f4963b045301d315db8f4 P 10883a25fd55430d9ced01c7987c7571: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:43:29.242766 15430 heartbeater.cc:503] Master 127.2.74.254:38849 was elected leader, sending a full tablet report...
I20260925 22:43:29.243719 15600 tablet_bootstrap.cc:654] T 890487a4220f4963b045301d315db8f4 P 9ec178f52e35455381d6e746b7675da3: Neither blocks nor log segments found. Creating new log.
I20260925 22:43:29.244669 15600 log.cc:824] T 890487a4220f4963b045301d315db8f4 P 9ec178f52e35455381d6e746b7675da3: Log is configured to *not* fsync() on all Append() calls
I20260925 22:43:29.247203 15598 tablet_bootstrap.cc:492] T 890487a4220f4963b045301d315db8f4 P cc548125786d48718d46eaceb3f0a3bf: No bootstrap required, opened a new log
I20260925 22:43:29.247527 15598 ts_tablet_manager.cc:1397] T 890487a4220f4963b045301d315db8f4 P cc548125786d48718d46eaceb3f0a3bf: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.005s
I20260925 22:43:29.248884 15600 tablet_bootstrap.cc:492] T 890487a4220f4963b045301d315db8f4 P 9ec178f52e35455381d6e746b7675da3: No bootstrap required, opened a new log
I20260925 22:43:29.249089 15600 ts_tablet_manager.cc:1397] T 890487a4220f4963b045301d315db8f4 P 9ec178f52e35455381d6e746b7675da3: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260925 22:43:29.252255 15598 raft_consensus.cc:348] T 890487a4220f4963b045301d315db8f4 P cc548125786d48718d46eaceb3f0a3bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc548125786d48718d46eaceb3f0a3bf" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44019 } } peers { permanent_uuid: "9ec178f52e35455381d6e746b7675da3" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36405 } } peers { permanent_uuid: "10883a25fd55430d9ced01c7987c7571" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 42495 } }
I20260925 22:43:29.252707 15598 raft_consensus.cc:374] T 890487a4220f4963b045301d315db8f4 P cc548125786d48718d46eaceb3f0a3bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:43:29.252811 15598 raft_consensus.cc:729] T 890487a4220f4963b045301d315db8f4 P cc548125786d48718d46eaceb3f0a3bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc548125786d48718d46eaceb3f0a3bf, State: Initialized, Role: FOLLOWER
I20260925 22:43:29.253185 15600 raft_consensus.cc:348] T 890487a4220f4963b045301d315db8f4 P 9ec178f52e35455381d6e746b7675da3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc548125786d48718d46eaceb3f0a3bf" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44019 } } peers { permanent_uuid: "9ec178f52e35455381d6e746b7675da3" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36405 } } peers { permanent_uuid: "10883a25fd55430d9ced01c7987c7571" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 42495 } }
I20260925 22:43:29.253472 15600 raft_consensus.cc:374] T 890487a4220f4963b045301d315db8f4 P 9ec178f52e35455381d6e746b7675da3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:43:29.253536 15600 raft_consensus.cc:729] T 890487a4220f4963b045301d315db8f4 P 9ec178f52e35455381d6e746b7675da3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ec178f52e35455381d6e746b7675da3, State: Initialized, Role: FOLLOWER
I20260925 22:43:29.253618 15598 consensus_queue.cc:260] T 890487a4220f4963b045301d315db8f4 P cc548125786d48718d46eaceb3f0a3bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc548125786d48718d46eaceb3f0a3bf" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44019 } } peers { permanent_uuid: "9ec178f52e35455381d6e746b7675da3" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36405 } } peers { permanent_uuid: "10883a25fd55430d9ced01c7987c7571" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 42495 } }
I20260925 22:43:29.254905 15598 ts_tablet_manager.cc:1428] T 890487a4220f4963b045301d315db8f4 P cc548125786d48718d46eaceb3f0a3bf: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:43:29.255374 15296 heartbeater.cc:503] Master 127.2.74.254:38849 was elected leader, sending a full tablet report...
I20260925 22:43:29.255280 15600 consensus_queue.cc:260] T 890487a4220f4963b045301d315db8f4 P 9ec178f52e35455381d6e746b7675da3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc548125786d48718d46eaceb3f0a3bf" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44019 } } peers { permanent_uuid: "9ec178f52e35455381d6e746b7675da3" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36405 } } peers { permanent_uuid: "10883a25fd55430d9ced01c7987c7571" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 42495 } }
I20260925 22:43:29.256536 15600 ts_tablet_manager.cc:1428] T 890487a4220f4963b045301d315db8f4 P 9ec178f52e35455381d6e746b7675da3: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 22:43:29.256963 15564 heartbeater.cc:503] Master 127.2.74.254:38849 was elected leader, sending a full tablet report...
W20260925 22:43:29.302116 15431 tablet.cc:2369] T 890487a4220f4963b045301d315db8f4 P 10883a25fd55430d9ced01c7987c7571: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:43:29.354218 15565 tablet.cc:2369] T 890487a4220f4963b045301d315db8f4 P 9ec178f52e35455381d6e746b7675da3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:43:29.422111 15605 raft_consensus.cc:482] T 890487a4220f4963b045301d315db8f4 P cc548125786d48718d46eaceb3f0a3bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:43:29.422358 15605 raft_consensus.cc:504] T 890487a4220f4963b045301d315db8f4 P cc548125786d48718d46eaceb3f0a3bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc548125786d48718d46eaceb3f0a3bf" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44019 } } peers { permanent_uuid: "9ec178f52e35455381d6e746b7675da3" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36405 } } peers { permanent_uuid: "10883a25fd55430d9ced01c7987c7571" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 42495 } }
I20260925 22:43:29.423820 15605 leader_election.cc:288] T 890487a4220f4963b045301d315db8f4 P cc548125786d48718d46eaceb3f0a3bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ec178f52e35455381d6e746b7675da3 (127.2.74.195:36405), 10883a25fd55430d9ced01c7987c7571 (127.2.74.194:42495)
I20260925 22:43:29.429394 15595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:29.425097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:57669 (local address 127.2.74.195:36405)
0925 22:43:29.425217 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:29.425221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:29.425234 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:43:29.425288 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:29.425290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:29.425328 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:29.425417 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:43:29.425756 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:29.426503 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:29.428085 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:29.428664 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:29.428752 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:29.428802 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:29.429145 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 22:43:29.429222 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:43:29.429840 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:29.424233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:36405 (local address 127.2.74.193:57669)
0925 22:43:29.424977 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:29.424992 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:29.425104 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:29.425435 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:29.425438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:29.425457 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:43:29.425616 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:29.425625 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:29.426631 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:29.426634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:29.427945 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:29.427955 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:29.428891 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:29.428894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:29.429004 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:29.429027 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:43:29.429575 (+ 548us) client_negotiation.cc:240] Negotiation successful
0925 22:43:29.429648 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":124,"threads_started":1}
I20260925 22:43:29.430044 15597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:29.424836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:49849 (local address 127.2.74.194:42495)
0925 22:43:29.424980 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:29.424985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:29.425779 (+ 794us) server_negotiation.cc:407] Connection header received
0925 22:43:29.425952 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:29.425955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:29.426009 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:29.426055 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:43:29.426431 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:29.427156 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:29.428530 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:29.429097 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:29.429179 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:29.429221 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:29.429829 (+ 608us) server_negotiation.cc:299] Negotiation successful
0925 22:43:29.429923 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:43:29.430850 15517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "890487a4220f4963b045301d315db8f4" candidate_uuid: "cc548125786d48718d46eaceb3f0a3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ec178f52e35455381d6e746b7675da3" is_pre_election: true
I20260925 22:43:29.431214 15517 raft_consensus.cc:2454] T 890487a4220f4963b045301d315db8f4 P 9ec178f52e35455381d6e746b7675da3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc548125786d48718d46eaceb3f0a3bf in term 0.
I20260925 22:43:29.431744 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:29.424689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:42495 (local address 127.2.74.193:49849)
0925 22:43:29.425702 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:29.425717 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:29.425842 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:29.426182 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:29.426185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:29.426200 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:43:29.426308 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:29.426314 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:29.427290 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:29.427293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:29.428392 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:29.428400 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:29.429306 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:29.429309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:29.429395 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:29.429434 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 22:43:29.431519 (+ 2085us) client_negotiation.cc:240] Negotiation successful
0925 22:43:29.431610 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"thread_start_us":113,"threads_started":1}
I20260925 22:43:29.431758 15183 leader_election.cc:302] T 890487a4220f4963b045301d315db8f4 P cc548125786d48718d46eaceb3f0a3bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ec178f52e35455381d6e746b7675da3, cc548125786d48718d46eaceb3f0a3bf; no voters:
I20260925 22:43:29.432255 15605 raft_consensus.cc:2781] T 890487a4220f4963b045301d315db8f4 P cc548125786d48718d46eaceb3f0a3bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:43:29.432344 15605 raft_consensus.cc:482] T 890487a4220f4963b045301d315db8f4 P cc548125786d48718d46eaceb3f0a3bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:43:29.432403 15605 raft_consensus.cc:3037] T 890487a4220f4963b045301d315db8f4 P cc548125786d48718d46eaceb3f0a3bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:43:29.433856 15382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "890487a4220f4963b045301d315db8f4" candidate_uuid: "cc548125786d48718d46eaceb3f0a3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10883a25fd55430d9ced01c7987c7571" is_pre_election: true
I20260925 22:43:29.434190 15382 raft_consensus.cc:2454] T 890487a4220f4963b045301d315db8f4 P 10883a25fd55430d9ced01c7987c7571 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc548125786d48718d46eaceb3f0a3bf in term 0.
I20260925 22:43:29.433431 15605 raft_consensus.cc:504] T 890487a4220f4963b045301d315db8f4 P cc548125786d48718d46eaceb3f0a3bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc548125786d48718d46eaceb3f0a3bf" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44019 } } peers { permanent_uuid: "9ec178f52e35455381d6e746b7675da3" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36405 } } peers { permanent_uuid: "10883a25fd55430d9ced01c7987c7571" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 42495 } }
I20260925 22:43:29.435491 15605 leader_election.cc:288] T 890487a4220f4963b045301d315db8f4 P cc548125786d48718d46eaceb3f0a3bf [CANDIDATE]: Term 1 election: Requested vote from peers 9ec178f52e35455381d6e746b7675da3 (127.2.74.195:36405), 10883a25fd55430d9ced01c7987c7571 (127.2.74.194:42495)
I20260925 22:43:29.435869 15517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "890487a4220f4963b045301d315db8f4" candidate_uuid: "cc548125786d48718d46eaceb3f0a3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ec178f52e35455381d6e746b7675da3"
I20260925 22:43:29.436039 15517 raft_consensus.cc:3037] T 890487a4220f4963b045301d315db8f4 P 9ec178f52e35455381d6e746b7675da3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:43:29.437139 15517 raft_consensus.cc:2454] T 890487a4220f4963b045301d315db8f4 P 9ec178f52e35455381d6e746b7675da3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc548125786d48718d46eaceb3f0a3bf in term 1.
I20260925 22:43:29.437460 15382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "890487a4220f4963b045301d315db8f4" candidate_uuid: "cc548125786d48718d46eaceb3f0a3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10883a25fd55430d9ced01c7987c7571"
I20260925 22:43:29.437510 15183 leader_election.cc:302] T 890487a4220f4963b045301d315db8f4 P cc548125786d48718d46eaceb3f0a3bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ec178f52e35455381d6e746b7675da3, cc548125786d48718d46eaceb3f0a3bf; no voters:
I20260925 22:43:29.437593 15382 raft_consensus.cc:3037] T 890487a4220f4963b045301d315db8f4 P 10883a25fd55430d9ced01c7987c7571 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:43:29.437709 15605 raft_consensus.cc:2781] T 890487a4220f4963b045301d315db8f4 P cc548125786d48718d46eaceb3f0a3bf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:43:29.437947 15605 raft_consensus.cc:686] T 890487a4220f4963b045301d315db8f4 P cc548125786d48718d46eaceb3f0a3bf [term 1 LEADER]: Becoming Leader. State: Replica: cc548125786d48718d46eaceb3f0a3bf, State: Running, Role: LEADER
I20260925 22:43:29.438298 15605 consensus_queue.cc:237] T 890487a4220f4963b045301d315db8f4 P cc548125786d48718d46eaceb3f0a3bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc548125786d48718d46eaceb3f0a3bf" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44019 } } peers { permanent_uuid: "9ec178f52e35455381d6e746b7675da3" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36405 } } peers { permanent_uuid: "10883a25fd55430d9ced01c7987c7571" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 42495 } }
I20260925 22:43:29.438861 15382 raft_consensus.cc:2454] T 890487a4220f4963b045301d315db8f4 P 10883a25fd55430d9ced01c7987c7571 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc548125786d48718d46eaceb3f0a3bf in term 1.
I20260925 22:43:29.442582 15098 catalog_manager.cc:5495] T 890487a4220f4963b045301d315db8f4 P cc548125786d48718d46eaceb3f0a3bf reported cstate change: term changed from 0 to 1, leader changed from <none> to cc548125786d48718d46eaceb3f0a3bf (127.2.74.193). New cstate: current_term: 1 leader_uuid: "cc548125786d48718d46eaceb3f0a3bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc548125786d48718d46eaceb3f0a3bf" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 44019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ec178f52e35455381d6e746b7675da3" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10883a25fd55430d9ced01c7987c7571" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 42495 } health_report { overall_health: UNKNOWN } } }
I20260925 22:43:29.488973 15594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:29.483338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35186 (local address 127.2.74.193:44019)
0925 22:43:29.483457 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:29.483465 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:29.483482 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:43:29.483569 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:29.483572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:29.483629 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:29.483767 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:43:29.484230 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:29.485057 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:29.486460 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:29.486818 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:29.486939 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:29.487504 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:43:29.488131 (+ 627us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:43:29.488272 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:29.488711 (+ 439us) server_negotiation.cc:299] Negotiation successful
0925 22:43:29.488788 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 22:43:29.494789 15297 tablet.cc:2369] T 890487a4220f4963b045301d315db8f4 P cc548125786d48718d46eaceb3f0a3bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:43:29.503852 15382 raft_consensus.cc:1261] T 890487a4220f4963b045301d315db8f4 P 10883a25fd55430d9ced01c7987c7571 [term 1 FOLLOWER]: Refusing update from remote peer cc548125786d48718d46eaceb3f0a3bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:43:29.504812 15605 consensus_queue.cc:1035] T 890487a4220f4963b045301d315db8f4 P cc548125786d48718d46eaceb3f0a3bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "10883a25fd55430d9ced01c7987c7571" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 42495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:43:29.505501 15517 raft_consensus.cc:1261] T 890487a4220f4963b045301d315db8f4 P 9ec178f52e35455381d6e746b7675da3 [term 1 FOLLOWER]: Refusing update from remote peer cc548125786d48718d46eaceb3f0a3bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:43:29.506229 15605 consensus_queue.cc:1035] T 890487a4220f4963b045301d315db8f4 P cc548125786d48718d46eaceb3f0a3bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ec178f52e35455381d6e746b7675da3" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:43:29.518906 15614 mvcc.cc:205] Tried to move back new op lower bound from 7333380954117312512 to 7333380953865498624. Current Snapshot: MvccSnapshot[applied={T|T < 7333380954117312512}]
I20260925 22:43:29.527127 15615 mvcc.cc:205] Tried to move back new op lower bound from 7333380954117312512 to 7333380953865498624. Current Snapshot: MvccSnapshot[applied={T|T < 7333380954117312512}]
I20260925 22:43:29.529596 15098 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:43:29.529793 15098 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:43:29.532351 15098 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 110b9b188a944147953c1a74fe93cb9e: Sending DeleteTablet for 3 replicas of tablet 890487a4220f4963b045301d315db8f4
I20260925 22:43:29.534034 15493 tablet_service.cc:1514] Processing DeleteTablet for tablet 890487a4220f4963b045301d315db8f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:43:29 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:44554
I20260925 22:43:29.534315 15230 tablet_service.cc:1514] Processing DeleteTablet for tablet 890487a4220f4963b045301d315db8f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:43:29 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:35184
I20260925 22:43:29.534585 15364 tablet_service.cc:1514] Processing DeleteTablet for tablet 890487a4220f4963b045301d315db8f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:43:29 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:60468
I20260925 22:43:29.535152 15624 tablet_replica.cc:330] T 890487a4220f4963b045301d315db8f4 P 10883a25fd55430d9ced01c7987c7571: stopping tablet replica
I20260925 22:43:29.535548 15624 raft_consensus.cc:2229] T 890487a4220f4963b045301d315db8f4 P 10883a25fd55430d9ced01c7987c7571 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:43:29.535817 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 15136
I20260925 22:43:29.535975 15624 raft_consensus.cc:2258] T 890487a4220f4963b045301d315db8f4 P 10883a25fd55430d9ced01c7987c7571 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:43:29.538431 15624 ts_tablet_manager.cc:1905] T 890487a4220f4963b045301d315db8f4 P 10883a25fd55430d9ced01c7987c7571: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:43:29.540426 15622 tablet_replica.cc:330] T 890487a4220f4963b045301d315db8f4 P 9ec178f52e35455381d6e746b7675da3: stopping tablet replica
I20260925 22:43:29.540889 15622 raft_consensus.cc:2229] T 890487a4220f4963b045301d315db8f4 P 9ec178f52e35455381d6e746b7675da3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:43:29.541364 15622 raft_consensus.cc:2258] T 890487a4220f4963b045301d315db8f4 P 9ec178f52e35455381d6e746b7675da3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:43:29.542233 15624 ts_tablet_manager.cc:1918] T 890487a4220f4963b045301d315db8f4 P 10883a25fd55430d9ced01c7987c7571: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:43:29.542408 15624 log.cc:1196] T 890487a4220f4963b045301d315db8f4 P 10883a25fd55430d9ced01c7987c7571: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-1/wal/wals/890487a4220f4963b045301d315db8f4
I20260925 22:43:29.542873 15624 ts_tablet_manager.cc:1939] T 890487a4220f4963b045301d315db8f4 P 10883a25fd55430d9ced01c7987c7571: Deleting consensus metadata
I20260925 22:43:29.543030 15622 ts_tablet_manager.cc:1905] T 890487a4220f4963b045301d315db8f4 P 9ec178f52e35455381d6e746b7675da3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:43:29.544500 15084 catalog_manager.cc:4841] TS 10883a25fd55430d9ced01c7987c7571 (127.2.74.194:42495): tablet 890487a4220f4963b045301d315db8f4 (table test-table [id=50b9150d871144cd95e67216b0020fbb]) successfully deleted
I20260925 22:43:29.545893 15622 ts_tablet_manager.cc:1918] T 890487a4220f4963b045301d315db8f4 P 9ec178f52e35455381d6e746b7675da3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:43:29.546011 15622 log.cc:1196] T 890487a4220f4963b045301d315db8f4 P 9ec178f52e35455381d6e746b7675da3: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376047580784-2347-0/minicluster-data/ts-2/wal/wals/890487a4220f4963b045301d315db8f4
I20260925 22:43:29.546327 15622 ts_tablet_manager.cc:1939] T 890487a4220f4963b045301d315db8f4 P 9ec178f52e35455381d6e746b7675da3: Deleting consensus metadata
I20260925 22:43:29.548642 15082 catalog_manager.cc:4841] TS 9ec178f52e35455381d6e746b7675da3 (127.2.74.195:36405): tablet 890487a4220f4963b045301d315db8f4 (table test-table [id=50b9150d871144cd95e67216b0020fbb]) successfully deleted
W20260925 22:43:29.554054 15085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.74.193:44019 (error 108)
I20260925 22:43:29.555135 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 15298
I20260925 22:43:29.564062 15596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:29.563671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:44019 (local address 127.0.0.1:35190)
0925 22:43:29.563850 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:29.563937 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.193:44019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 22:43:29.564281 15085 catalog_manager.cc:4584] TS cc548125786d48718d46eaceb3f0a3bf (127.2.74.193:44019): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 890487a4220f4963b045301d315db8f4: Network error: Client connection negotiation failed: client connection to 127.2.74.193:44019: connect: Connection refused (error 111)
I20260925 22:43:29.564528 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 15433
I20260925 22:43:29.570330 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 15065
I20260925 22:43:29.579474 2347 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 22:43:29.915499 2347 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 22:43:29.915599 2347 mini_ranger.cc:66] Stopping Ranger...
I20260925 22:43:30.261737 2347 mini_ranger.cc:68] Stopped Ranger
2026-09-25 22:43:30.261 UTC [14132] LOG: received smart shutdown request
2026-09-25 22:43:30.265 UTC [14132] LOG: background worker "logical replication launcher" (PID 14141) exited with exit code 1
2026-09-25 22:43:30.266 UTC [14136] LOG: shutting down
2026-09-25 22:43:30.450 UTC [14132] LOG: database system is shut down
2026-09-25T22:43:30Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51972 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:43:30 dist-test-slave-csnl krb5kdc[15636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:43:30 dist-test-slave-csnl krb5kdc[15636](info): set up 2 sockets
Sep 25 22:43:30 dist-test-slave-csnl krb5kdc[15636](info): commencing operation
krb5kdc: starting...
W20260925 22:43:31.627144 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:43:31 dist-test-slave-csnl krb5kdc[15636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:43:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:43:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:43:31.754732 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:36829
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:43233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:36829,127.2.74.253:43483,127.2.74.252:35083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:43:31.869204 15649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:43:31.869566 15649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:43:31.869660 15649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:43:31.873974 15649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:43:31.874100 15649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:43:31.874140 15649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:43:31.874181 15649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:43:31.874212 15649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:43:31.879999 15649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:43233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:36829,127.2.74.253:43483,127.2.74.252:35083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:36829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:43:31.881271 15649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:43:31.882429 15649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:43:31.891635 15657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:31.891635 15654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:31.891736 15655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:31.892081 15649 server_base.cc:1034] running on GCE node
I20260925 22:43:31.892704 15649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:43:31.893953 15649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:43:31.895123 15649 hybrid_clock.cc:648] HybridClock initialized: now 1790376211895090 us; error 52 us; skew 500 ppm
Sep 25 22:43:31 dist-test-slave-csnl krb5kdc[15636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376211, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:43:31.898111 15649 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:43:31.899300 15649 webserver.cc:460] Webserver started at http://127.2.74.254:42307/ using document root <none> and password file <none>
I20260925 22:43:31.899971 15649 fs_manager.cc:359] Metadata directory not provided
I20260925 22:43:31.900081 15649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:43:31.900336 15649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:43:31.902597 15649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "3ba589dbaae049e2bcd69d876db86c64"
format_stamp: "Formatted at 2026-09-25 22:43:31 on dist-test-slave-csnl"
server_key: "08258b04306931fbe8e9a721020f4386"
server_key_iv: "359a674f38227a4d73c468ff1384696f"
server_key_version: "encryptionkey@0"
I20260925 22:43:31.903280 15649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "3ba589dbaae049e2bcd69d876db86c64"
format_stamp: "Formatted at 2026-09-25 22:43:31 on dist-test-slave-csnl"
server_key: "08258b04306931fbe8e9a721020f4386"
server_key_iv: "359a674f38227a4d73c468ff1384696f"
server_key_version: "encryptionkey@0"
I20260925 22:43:31.907424 15649 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:43:31.910025 15664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:31.911453 15649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:43:31.911613 15649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "3ba589dbaae049e2bcd69d876db86c64"
format_stamp: "Formatted at 2026-09-25 22:43:31 on dist-test-slave-csnl"
server_key: "08258b04306931fbe8e9a721020f4386"
server_key_iv: "359a674f38227a4d73c468ff1384696f"
server_key_version: "encryptionkey@0"
I20260925 22:43:31.911746 15649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:43:31.923969 15649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:43:31.927105 15649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:43:31.927345 15649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:43:31.934533 15649 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:36829
I20260925 22:43:31.934544 15716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:36829 every 8 connection(s)
I20260925 22:43:31.935847 15649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:43:31.938748 15717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:43:31.941659 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 15649
I20260925 22:43:31.941823 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:43:31.942114 2347 external_mini_cluster.cc:1428] Setting key 220fa12e1a431bd1c2c38d0b282569ac
I20260925 22:43:31.942736 15717 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36829 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.74.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.253@KRBTEST.COM" created.
Sep 25 22:43:31 dist-test-slave-csnl krb5kdc[15636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376211, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:43:31.959780 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:31.945369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:36829 (local address 127.0.0.1:35470)
0925 22:43:31.945990 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:31.946040 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:31.946900 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:31.947899 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:31.947904 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:31.948238 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:43:31.948558 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:31.948571 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:31.950393 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:31.950397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:31.951493 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:31.951503 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:31.952204 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:31.952216 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:31.952308 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:31.952942 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:43:31.952958 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:43:31.954552 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:43:31.956537 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:31.956547 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:31.956559 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:31.956790 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:31.957139 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:31.957142 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:31.957144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:31.957261 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:31.957872 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:43:31.957879 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:43:31.958047 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 22:43:31.958254 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 22:43:31.958613 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"spinlock_wait_cycles":31616,"thread_start_us":176,"threads_started":1}
I20260925 22:43:31.959784 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:31.945369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35470 (local address 127.2.74.254:36829)
0925 22:43:31.945896 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:31.945916 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:31.946167 (+ 251us) server_negotiation.cc:407] Connection header received
0925 22:43:31.947319 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:31.947329 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:31.947637 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:31.947933 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:31.948723 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:31.950226 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:31.951607 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:31.952038 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:31.952165 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:31.954708 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:31.954722 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:31.954733 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:31.954754 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:31.956406 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:31.956909 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:31.956922 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:31.956924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:31.957035 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:31.957353 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:31.957359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:31.957361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:31.957756 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:31.957903 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:31.958355 (+ 452us) server_negotiation.cc:299] Negotiation successful
0925 22:43:31.958711 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":342,"thread_start_us":137,"threads_started":1}
I20260925 22:43:31.963500 15717 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.253" port: 43483 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:43:31.964329 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:31.964008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:43483 (local address 127.0.0.1:51954)
0925 22:43:31.964177 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:31.964255 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.253:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260925 22:43:31.964552 15666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.74.253:43483: connect: Connection refused (error 111)
I20260925 22:43:31.965572 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:31.965278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:43483 (local address 127.0.0.1:51966)
0925 22:43:31.965426 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:31.965510 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.253:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 22:43:31.965728 15717 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.74.253:43483: Network error: Client connection negotiation failed: client connection to 127.2.74.253:43483: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.253.keytab.
Entry for principal kudu/127.2.74.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.253.keytab.
I20260925 22:43:31.969833 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.253:43483
--webserver_interface=127.2.74.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.253.keytab
--principal=kudu/127.2.74.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:43233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:36829,127.2.74.253:43483,127.2.74.252:35083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:43:32.015019 15717 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.74.253" port: 43483 } attempt: 1
I20260925 22:43:32.015941 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.015602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:43483 (local address 127.0.0.1:51980)
0925 22:43:32.015747 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:32.015837 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.253:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 22:43:32.016708 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.016469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:43483 (local address 127.0.0.1:51988)
0925 22:43:32.016577 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:32.016631 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.253:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 22:43:32.016841 15717 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.74.253:43483: Network error: Client connection negotiation failed: client connection to 127.2.74.253:43483: connect: Connection refused (error 111)
I20260925 22:43:32.085065 15717 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.74.253" port: 43483 } attempt: 2
I20260925 22:43:32.086004 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.085625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:43483 (local address 127.0.0.1:52000)
0925 22:43:32.085843 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:32.085918 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.253:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260925 22:43:32.086119 15724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:43:32.086561 15724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:43:32.086632 15724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 22:43:32.086925 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.086656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:43483 (local address 127.0.0.1:52014)
0925 22:43:32.086803 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:32.086858 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.253:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 22:43:32.087113 15717 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.74.253:43483: Network error: Client connection negotiation failed: client connection to 127.2.74.253:43483: connect: Connection refused (error 111)
W20260925 22:43:32.091346 15724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:43:32.091413 15724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:43:32.091434 15724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:43:32.091450 15724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:43:32.091502 15724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:43:32.097121 15724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:43233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:36829,127.2.74.253:43483,127.2.74.252:35083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.253.keytab
--principal=kudu/127.2.74.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.253:43483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.74.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:43:32.098278 15724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:43:32.099404 15724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:43:32.107980 15732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:32.107996 15729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:32.107980 15730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:32.108304 15724 server_base.cc:1034] running on GCE node
I20260925 22:43:32.108884 15724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:43:32.110057 15724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:43:32.111251 15724 hybrid_clock.cc:648] HybridClock initialized: now 1790376212111234 us; error 31 us; skew 500 ppm
Sep 25 22:43:32 dist-test-slave-csnl krb5kdc[15636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376212, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:43:32.114005 15724 init.cc:377] Logged in from keytab as kudu/127.2.74.253@KRBTEST.COM (short username kudu)
I20260925 22:43:32.114929 15724 webserver.cc:460] Webserver started at http://127.2.74.253:36265/ using document root <none> and password file <none>
I20260925 22:43:32.115633 15724 fs_manager.cc:359] Metadata directory not provided
I20260925 22:43:32.115726 15724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:43:32.115957 15724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:43:32.118046 15724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-1/data/instance:
uuid: "de70d0a3b83b4d0aa286fb606079b831"
format_stamp: "Formatted at 2026-09-25 22:43:32 on dist-test-slave-csnl"
server_key: "04fd2c57d6caf4996ef7a43780525356"
server_key_iv: "7d30f0e887844307dda4b4fd2f7137d0"
server_key_version: "encryptionkey@0"
I20260925 22:43:32.118633 15724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-1/wal/instance:
uuid: "de70d0a3b83b4d0aa286fb606079b831"
format_stamp: "Formatted at 2026-09-25 22:43:32 on dist-test-slave-csnl"
server_key: "04fd2c57d6caf4996ef7a43780525356"
server_key_iv: "7d30f0e887844307dda4b4fd2f7137d0"
server_key_version: "encryptionkey@0"
I20260925 22:43:32.122402 15724 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260925 22:43:32.125103 15739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:32.126396 15724 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:43:32.126549 15724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-1/wal
uuid: "de70d0a3b83b4d0aa286fb606079b831"
format_stamp: "Formatted at 2026-09-25 22:43:32 on dist-test-slave-csnl"
server_key: "04fd2c57d6caf4996ef7a43780525356"
server_key_iv: "7d30f0e887844307dda4b4fd2f7137d0"
server_key_version: "encryptionkey@0"
I20260925 22:43:32.126673 15724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:43:32.137889 15724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:43:32.141029 15724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:43:32.141239 15724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:43:32.148654 15724 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.253:43483
I20260925 22:43:32.148674 15791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.253:43483 every 8 connection(s)
I20260925 22:43:32.149705 15724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-1/data/info.pb
I20260925 22:43:32.152632 15792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:43:32.156363 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 15724
I20260925 22:43:32.156528 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-1/wal/instance
I20260925 22:43:32.156800 2347 external_mini_cluster.cc:1428] Setting key 2ed7067dfce0deb344dd8e1daa78797c
I20260925 22:43:32.156860 15792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36829 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.74.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.252@KRBTEST.COM" created.
Sep 25 22:43:32 dist-test-slave-csnl krb5kdc[15636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376212, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.253@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:43:32.173153 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.158911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35480 (local address 127.2.74.254:36829)
0925 22:43:32.159100 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:32.159105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:32.160114 (+ 1009us) server_negotiation.cc:407] Connection header received
0925 22:43:32.161095 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:32.161100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:32.161174 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:32.161288 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:32.162480 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.163395 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.164582 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.165023 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.165117 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:32.168316 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:32.168320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:32.168322 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:32.168351 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:32.170716 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:32.171472 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:32.171475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:32.171476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:32.171555 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:32.171956 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:32.171959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:32.171960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:32.172153 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:32.172263 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:32.172832 (+ 569us) server_negotiation.cc:299] Negotiation successful
0925 22:43:32.172975 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 22:43:32.174014 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.159366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:36829 (local address 127.0.0.1:35480)
0925 22:43:32.159924 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:32.159981 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:32.160874 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:32.161457 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:32.161465 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:32.161814 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:43:32.162301 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:32.162315 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.163539 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.163542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:32.164445 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:32.164455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.165171 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.165174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:32.165343 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:32.166059 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:43:32.166076 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:43:32.168111 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:43:32.170867 (+ 2756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:32.170878 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:32.170901 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:32.171355 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:32.171661 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:32.171665 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:32.171667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:32.171837 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:32.172263 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:43:32.172271 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:43:32.172558 (+ 287us) client_negotiation.cc:769] Sending connection context
0925 22:43:32.172758 (+ 200us) client_negotiation.cc:240] Negotiation successful
0925 22:43:32.173129 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":157,"threads_started":1}
I20260925 22:43:32.175489 15792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.253" port: 43483 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:43:32.178284 15717 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.74.253" port: 43483 } attempt: 3
Sep 25 22:43:32 dist-test-slave-csnl krb5kdc[15636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376212, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.253@KRBTEST.COM for kudu/127.2.74.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.252.keytab.
Entry for principal kudu/127.2.74.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.252.keytab.
Sep 25 22:43:32 dist-test-slave-csnl krb5kdc[15636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376211, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for kudu/127.2.74.253@KRBTEST.COM
I20260925 22:43:32.184334 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.252:35083
--webserver_interface=127.2.74.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.252.keytab
--principal=kudu/127.2.74.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:43233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:36829,127.2.74.253:43483,127.2.74.252:35083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:43:32.186654 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.176278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52020 (local address 127.2.74.253:43483)
0925 22:43:32.176577 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:32.176586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:32.176606 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:43:32.176642 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:32.176647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:32.176801 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:32.176962 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:32.177280 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.178124 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.179126 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.179671 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.179886 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:32.182192 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:32.182200 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:32.182214 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:32.182231 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:32.184425 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:32.184951 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:32.184961 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:32.184963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:32.185088 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:32.185441 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:32.185445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:32.185448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:32.185816 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:32.185955 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:32.186241 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 22:43:32.186408 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":209,"thread_start_us":109,"threads_started":1}
I20260925 22:43:32.187153 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.176178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:43483 (local address 127.0.0.1:52020)
0925 22:43:32.176313 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:32.176346 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:32.176431 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:32.176949 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:32.176951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:32.177088 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:43:32.177182 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:32.177188 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.178242 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.178245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:32.179015 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:32.179024 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.179804 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.179811 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:32.179998 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:32.180494 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:43:32.180506 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:43:32.182080 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:43:32.184550 (+ 2470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:32.184556 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:32.184561 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:32.184821 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:32.185208 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:32.185214 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:32.185219 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:32.185333 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:32.185914 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:43:32.185917 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:43:32.186009 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 22:43:32.186813 (+ 804us) client_negotiation.cc:240] Negotiation successful
0925 22:43:32.186982 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":78}
I20260925 22:43:32.188476 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.178778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:43483 (local address 127.0.0.1:52028)
0925 22:43:32.178907 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:32.178922 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:32.179039 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:32.179423 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:32.179426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:32.179583 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:43:32.179685 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:32.179691 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.180643 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.180648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:32.181362 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:32.181372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.182014 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.182020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:32.182139 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:32.182687 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:43:32.182699 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:43:32.184171 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:43:32.186885 (+ 2714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:32.186888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:32.186890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:32.187175 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:32.187531 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:32.187539 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:32.187540 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:32.187624 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:32.188057 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:43:32.188061 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:43:32.188169 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 22:43:32.188256 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 22:43:32.188365 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 22:43:32.189073 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.178781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52028 (local address 127.2.74.253:43483)
0925 22:43:32.179156 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:32.179165 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:32.179177 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:43:32.179205 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:32.179208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:32.179257 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:32.179344 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:32.179820 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.180529 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.181489 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.181893 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.182026 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:32.184322 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:32.184326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:32.184328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:32.184362 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:32.186758 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:32.187307 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:32.187311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:32.187319 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:32.187424 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:32.187727 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:32.187730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:32.187732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:32.187923 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:32.188780 (+ 857us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:32.188853 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 22:43:32.188942 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":312,"thread_start_us":106,"threads_started":1}
I20260925 22:43:32.189664 15792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.252" port: 35083 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:43:32.189930 15717 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.252" port: 35083 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:43:32.190459 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.190156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:35083 (local address 127.0.0.1:32892)
0925 22:43:32.190334 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:32.190402 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260925 22:43:32.190668 15741 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.74.252:35083: connect: Connection refused (error 111)
I20260925 22:43:32.190876 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.190446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:35083 (local address 127.0.0.1:32898)
0925 22:43:32.190714 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:32.190776 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260925 22:43:32.191849 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.191544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:35083 (local address 127.0.0.1:32914)
0925 22:43:32.191697 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:32.191766 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 22:43:32.192042 15792 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.74.252:35083: Network error: Client connection negotiation failed: client connection to 127.2.74.252:35083: connect: Connection refused (error 111)
I20260925 22:43:32.192243 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.191551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:35083 (local address 127.0.0.1:32924)
0925 22:43:32.192091 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:32.192157 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451}
W20260925 22:43:32.192462 15717 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.74.252:35083: Network error: Client connection negotiation failed: client connection to 127.2.74.252:35083: connect: Connection refused (error 111)
I20260925 22:43:32.223652 15717 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.74.252" port: 35083 } attempt: 1
I20260925 22:43:32.224609 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.224236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:35083 (local address 127.0.0.1:32932)
0925 22:43:32.224423 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:32.224506 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 22:43:32.225474 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.225210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:35083 (local address 127.0.0.1:32936)
0925 22:43:32.225341 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:32.225400 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 22:43:32.225636 15717 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.74.252:35083: Network error: Client connection negotiation failed: client connection to 127.2.74.252:35083: connect: Connection refused (error 111)
I20260925 22:43:32.241339 15792 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.74.252" port: 35083 } attempt: 1
I20260925 22:43:32.242307 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.241934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:35083 (local address 127.0.0.1:32940)
0925 22:43:32.242116 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:32.242198 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 22:43:32.243180 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.242868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:35083 (local address 127.0.0.1:32946)
0925 22:43:32.243049 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:32.243110 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 22:43:32.243332 15792 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.74.252:35083: Network error: Client connection negotiation failed: client connection to 127.2.74.252:35083: connect: Connection refused (error 111)
I20260925 22:43:32.300838 15717 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.74.252" port: 35083 } attempt: 2
I20260925 22:43:32.301765 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.301414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:35083 (local address 127.0.0.1:32950)
0925 22:43:32.301589 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:32.301664 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 22:43:32.302597 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.302317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:35083 (local address 127.0.0.1:32958)
0925 22:43:32.302445 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:32.302511 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 22:43:32.302755 15717 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.74.252:35083: Network error: Client connection negotiation failed: client connection to 127.2.74.252:35083: connect: Connection refused (error 111)
W20260925 22:43:32.305691 15799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:43:32.305974 15799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:43:32.306016 15799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:43:32.310359 15799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:43:32.310443 15799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:43:32.310464 15799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:43:32.310482 15799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:43:32.310497 15799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:43:32.311480 15792 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.74.252" port: 35083 } attempt: 2
I20260925 22:43:32.312378 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.312003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:35083 (local address 127.0.0.1:32968)
0925 22:43:32.312177 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:32.312249 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 22:43:32.313136 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.312900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:35083 (local address 127.0.0.1:32982)
0925 22:43:32.313017 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:32.313068 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 22:43:32.313282 15792 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.74.252:35083: Network error: Client connection negotiation failed: client connection to 127.2.74.252:35083: connect: Connection refused (error 111)
I20260925 22:43:32.317157 15799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:43233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:36829,127.2.74.253:43483,127.2.74.252:35083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.252.keytab
--principal=kudu/127.2.74.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.252:35083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.74.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:43:32.318367 15799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:43:32.319585 15799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:43:32.328680 15806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:32.328640 15805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:32.328640 15808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:32.328985 15799 server_base.cc:1034] running on GCE node
I20260925 22:43:32.329763 15799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:43:32.331176 15799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:43:32.332379 15799 hybrid_clock.cc:648] HybridClock initialized: now 1790376212332356 us; error 41 us; skew 500 ppm
Sep 25 22:43:32 dist-test-slave-csnl krb5kdc[15636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376212, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:43:32.335284 15799 init.cc:377] Logged in from keytab as kudu/127.2.74.252@KRBTEST.COM (short username kudu)
I20260925 22:43:32.336319 15799 webserver.cc:460] Webserver started at http://127.2.74.252:45471/ using document root <none> and password file <none>
I20260925 22:43:32.336931 15799 fs_manager.cc:359] Metadata directory not provided
I20260925 22:43:32.337031 15799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:43:32.337275 15799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:43:32.339628 15799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-2/data/instance:
uuid: "d6feedda6f21464c94f23104a024808c"
format_stamp: "Formatted at 2026-09-25 22:43:32 on dist-test-slave-csnl"
server_key: "1fd2e9aec675251181606d6d9b48b47e"
server_key_iv: "90dd105b3cfb6178fd1616b7131d304c"
server_key_version: "encryptionkey@0"
I20260925 22:43:32.340248 15799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-2/wal/instance:
uuid: "d6feedda6f21464c94f23104a024808c"
format_stamp: "Formatted at 2026-09-25 22:43:32 on dist-test-slave-csnl"
server_key: "1fd2e9aec675251181606d6d9b48b47e"
server_key_iv: "90dd105b3cfb6178fd1616b7131d304c"
server_key_version: "encryptionkey@0"
I20260925 22:43:32.344906 15799 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:43:32.347646 15815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:32.349043 15799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:43:32.349224 15799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-2/wal
uuid: "d6feedda6f21464c94f23104a024808c"
format_stamp: "Formatted at 2026-09-25 22:43:32 on dist-test-slave-csnl"
server_key: "1fd2e9aec675251181606d6d9b48b47e"
server_key_iv: "90dd105b3cfb6178fd1616b7131d304c"
server_key_version: "encryptionkey@0"
I20260925 22:43:32.349380 15799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:43:32.389792 15799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:43:32.393385 15799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:43:32.393620 15799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:43:32.401420 15799 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.252:35083
I20260925 22:43:32.401436 15867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.252:35083 every 8 connection(s)
I20260925 22:43:32.401925 15717 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.74.252" port: 35083 } attempt: 3
I20260925 22:43:32.402647 15799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-2/data/info.pb
I20260925 22:43:32.403208 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 15799
I20260925 22:43:32.403374 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376047580784-2347-0/minicluster-data/master-2/wal/instance
I20260925 22:43:32.403759 2347 external_mini_cluster.cc:1428] Setting key 35f8c384ec5f0f3bab4a4747b1629e54
I20260925 22:43:32.404711 15792 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.74.252" port: 35083 } attempt: 3
I20260925 22:43:32.406641 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 22:43:32 dist-test-slave-csnl krb5kdc[15636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376211, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for kudu/127.2.74.252@KRBTEST.COM
Sep 25 22:43:32 dist-test-slave-csnl krb5kdc[15636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376212, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.253@KRBTEST.COM for kudu/127.2.74.252@KRBTEST.COM
I20260925 22:43:32.412591 15868 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36829 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:43:32 dist-test-slave-csnl krb5kdc[15636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.252@KRBTEST.COM
I20260925 22:43:32.418726 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.405132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:35083 (local address 127.0.0.1:33014)
0925 22:43:32.405359 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:32.405373 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:32.405450 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:32.406110 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:32.406113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:32.406286 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:43:32.406404 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:32.406412 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.408485 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.408489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:32.409365 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:32.409377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.410723 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.410727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:32.410825 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:32.411604 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:43:32.411623 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:43:32.413414 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:43:32.416819 (+ 3405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:32.416824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:32.416828 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:32.417112 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:32.417439 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:32.417442 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:32.417444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:32.417528 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:32.418208 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:43:32.418212 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:43:32.418313 (+ 101us) client_negotiation.cc:769] Sending connection context
0925 22:43:32.418400 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 22:43:32.418548 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260925 22:43:32.418726 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.402446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:35083 (local address 127.0.0.1:32998)
0925 22:43:32.402609 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:32.402628 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:32.402735 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:32.405863 (+ 3128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:32.405868 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:32.406113 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:43:32.406255 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:32.406266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.408486 (+ 2220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.408490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:32.409365 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:32.409378 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.410723 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.410727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:32.410825 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:32.411489 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:43:32.411502 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:43:32.413848 (+ 2346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:43:32.416527 (+ 2679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:32.416531 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:32.416533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:32.416808 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:32.417209 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:32.417212 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:32.417214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:32.417319 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:32.418209 (+ 890us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:43:32.418212 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:43:32.418313 (+ 101us) client_negotiation.cc:769] Sending connection context
0925 22:43:32.418414 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 22:43:32.418548 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 22:43:32.420804 15871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.404441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33000 (local address 127.2.74.252:35083)
0925 22:43:32.404758 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:32.404768 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:32.405048 (+ 280us) server_negotiation.cc:407] Connection header received
0925 22:43:32.405238 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:32.405241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:32.405508 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:32.405844 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:32.407262 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.408323 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.409518 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.410560 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.410729 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:32.415254 (+ 4525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:32.415259 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:32.415261 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:32.415290 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:32.418198 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:32.418705 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:32.418708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:32.418710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:32.418813 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:32.419282 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:32.419286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:32.419288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:32.419513 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:32.419653 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:32.420028 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 22:43:32.420161 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":376,"server-negotiator.queue_time_us":252,"spinlock_wait_cycles":8960,"thread_start_us":113,"threads_started":1}
I20260925 22:43:32.420796 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.403421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32998 (local address 127.2.74.252:35083)
0925 22:43:32.403943 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:32.403961 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:32.403998 (+ 37us) server_negotiation.cc:407] Connection header received
0925 22:43:32.404985 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:32.405013 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:32.405504 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:32.406391 (+ 887us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:32.406527 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.408313 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.409529 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.410560 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.411149 (+ 589us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:32.414016 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:32.414021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:32.414026 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:32.414059 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:32.416436 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:32.416928 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:32.416936 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:32.416943 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:32.417088 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:32.417442 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:32.417445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:32.417446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:32.418083 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:32.419128 (+ 1045us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:32.419329 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 22:43:32.419675 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":367,"server-negotiator.queue_time_us":286,"spinlock_wait_cycles":23936,"thread_start_us":140,"threads_started":1}
I20260925 22:43:32.420796 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.405575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33014 (local address 127.2.74.252:35083)
0925 22:43:32.405859 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:32.405869 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:32.405881 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:43:32.405912 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:32.405918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:32.405964 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:32.406509 (+ 545us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:32.406572 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.408313 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.409529 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.410560 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.411148 (+ 588us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:32.413639 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:32.413647 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:32.413665 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:32.413701 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:32.416688 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:32.417248 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:32.417253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:32.417258 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:32.417328 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:32.417639 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:32.417642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:32.417643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:32.418083 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:32.419508 (+ 1425us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:32.419577 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 22:43:32.419721 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":657,"server-negotiator.queue_time_us":209,"spinlock_wait_cycles":1792,"thread_start_us":84,"threads_started":1}
Sep 25 22:43:32 dist-test-slave-csnl krb5kdc[15636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376212, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.252@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:43:32.425945 15792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831: Bootstrap starting.
I20260925 22:43:32.425946 15717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ba589dbaae049e2bcd69d876db86c64: Bootstrap starting.
I20260925 22:43:32.428362 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.414931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:36829 (local address 127.0.0.1:35488)
0925 22:43:32.416157 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:32.416210 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:32.416334 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:32.416831 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:32.416839 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:32.417705 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:43:32.417873 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:32.417882 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.418934 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.418937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:32.419972 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:32.419980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.420673 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.420677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:32.420770 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:32.421423 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:43:32.421438 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:43:32.423870 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:43:32.426168 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:32.426181 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:32.426187 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:32.426559 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:32.426903 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:32.426906 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:32.426907 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:32.427055 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:32.427592 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:43:32.427600 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:43:32.427739 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 22:43:32.427878 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 22:43:32.428102 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1015,"thread_start_us":118,"threads_started":1}
I20260925 22:43:32.428993 15792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831: Neither blocks nor log segments found. Creating new log.
I20260925 22:43:32.429184 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.416210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35488 (local address 127.2.74.254:36829)
0925 22:43:32.416354 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:32.416361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:32.416378 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:43:32.416431 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:32.416436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:32.416506 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:32.416624 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:32.418026 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.418807 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.420095 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.420421 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.420515 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:32.424039 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:32.424044 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:32.424047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:32.424077 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:32.426015 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:32.426688 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:32.426691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:32.426693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:32.426798 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:32.427157 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:32.427161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:32.427162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:32.427440 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:32.427530 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:32.428826 (+ 1296us) server_negotiation.cc:299] Negotiation successful
0925 22:43:32.428951 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:43:32.428993 15717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ba589dbaae049e2bcd69d876db86c64: Neither blocks nor log segments found. Creating new log.
I20260925 22:43:32.429960 15792 log.cc:824] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831: Log is configured to *not* fsync() on all Append() calls
I20260925 22:43:32.430533 15717 log.cc:824] T 00000000000000000000000000000000 P 3ba589dbaae049e2bcd69d876db86c64: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:43:32 dist-test-slave-csnl krb5kdc[15636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:43:32.431348 15868 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.253" port: 43483 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:43:32.433292 15792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831: No bootstrap required, opened a new log
I20260925 22:43:32.433665 15717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ba589dbaae049e2bcd69d876db86c64: No bootstrap required, opened a new log
I20260925 22:43:32.436188 15875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.423451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35496 (local address 127.2.74.254:36829)
0925 22:43:32.424808 (+ 1357us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:32.424825 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:32.424839 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:43:32.424882 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:32.424886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:32.424947 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:32.426176 (+ 1229us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:32.426288 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.427178 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.428297 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.428655 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.431383 (+ 2728us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:32.431885 (+ 502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:32.431889 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:32.431892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:32.431917 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:32.433749 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:32.434323 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:32.434327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:32.434329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:32.434418 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:32.434749 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:32.434752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:32.434753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:32.435010 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:32.435767 (+ 757us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:32.435846 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 22:43:32.435974 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1261,"thread_start_us":75,"threads_started":1}
Sep 25 22:43:32 dist-test-slave-csnl krb5kdc[15636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376212, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.252@KRBTEST.COM for kudu/127.2.74.253@KRBTEST.COM
I20260925 22:43:32.437940 15792 raft_consensus.cc:348] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba589dbaae049e2bcd69d876db86c64" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36829 } } peers { permanent_uuid: "de70d0a3b83b4d0aa286fb606079b831" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 43483 } } peers { permanent_uuid: "d6feedda6f21464c94f23104a024808c" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 35083 } }
I20260925 22:43:32.438207 15792 raft_consensus.cc:374] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:43:32.438277 15792 raft_consensus.cc:729] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de70d0a3b83b4d0aa286fb606079b831, State: Initialized, Role: FOLLOWER
I20260925 22:43:32.438825 15792 consensus_queue.cc:260] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba589dbaae049e2bcd69d876db86c64" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36829 } } peers { permanent_uuid: "de70d0a3b83b4d0aa286fb606079b831" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 43483 } } peers { permanent_uuid: "d6feedda6f21464c94f23104a024808c" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 35083 } }
I20260925 22:43:32.439800 15717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ba589dbaae049e2bcd69d876db86c64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba589dbaae049e2bcd69d876db86c64" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36829 } } peers { permanent_uuid: "de70d0a3b83b4d0aa286fb606079b831" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 43483 } } peers { permanent_uuid: "d6feedda6f21464c94f23104a024808c" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 35083 } }
I20260925 22:43:32.440038 15717 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ba589dbaae049e2bcd69d876db86c64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:43:32.440094 15717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ba589dbaae049e2bcd69d876db86c64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba589dbaae049e2bcd69d876db86c64, State: Initialized, Role: FOLLOWER
I20260925 22:43:32.440514 15792 sys_catalog.cc:564] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:43:32.440651 15717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ba589dbaae049e2bcd69d876db86c64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba589dbaae049e2bcd69d876db86c64" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36829 } } peers { permanent_uuid: "de70d0a3b83b4d0aa286fb606079b831" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 43483 } } peers { permanent_uuid: "d6feedda6f21464c94f23104a024808c" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 35083 } }
I20260925 22:43:32.442445 15717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ba589dbaae049e2bcd69d876db86c64 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:43:32.442703 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.431942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:43483 (local address 127.0.0.1:52042)
0925 22:43:32.432191 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:32.432205 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:32.432283 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:32.432838 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:32.432841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:32.433040 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:43:32.433148 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:32.433155 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.434226 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.434229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:32.435087 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:32.435099 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.435818 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.435822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:32.435926 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:32.436697 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:43:32.436713 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:43:32.438347 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:43:32.440674 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:32.440678 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:32.440680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:32.440976 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:32.441428 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:32.441433 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:32.441438 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:32.441557 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:32.442112 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:43:32.442116 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:43:32.442236 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 22:43:32.442347 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 22:43:32.442512 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260925 22:43:32.439738 15878 sys_catalog.cc:455] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba589dbaae049e2bcd69d876db86c64" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36829 } } peers { permanent_uuid: "de70d0a3b83b4d0aa286fb606079b831" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 43483 } } peers { permanent_uuid: "d6feedda6f21464c94f23104a024808c" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 35083 } } }
I20260925 22:43:32.444097 15878 sys_catalog.cc:458] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831 [sys.catalog]: This master's current role is: FOLLOWER
Sep 25 22:43:32 dist-test-slave-csnl krb5kdc[15636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.253@KRBTEST.COM
I20260925 22:43:32.443720 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.432419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52042 (local address 127.2.74.253:43483)
0925 22:43:32.432540 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:32.432544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:32.432558 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:43:32.432599 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:32.432602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:32.432645 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:32.432718 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:32.433282 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.434053 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.435226 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.435570 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.435669 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:32.438678 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:32.438682 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:32.438684 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:32.438711 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:32.440539 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:32.441196 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:32.441199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:32.441201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:32.441296 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:32.441661 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:32.441665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:32.441666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:32.441952 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:32.442060 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:32.443410 (+ 1350us) server_negotiation.cc:299] Negotiation successful
0925 22:43:32.443541 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:43:32.445104 15868 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.252" port: 35083 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:43:32.445269 15724 master_runner.cc:186] Error getting master registration for 127.2.74.254:36829: OK, instance_id { permanent_uuid: "3ba589dbaae049e2bcd69d876db86c64" instance_seqno: 1790376211932488 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:43:32.448649 15724 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:43:32.449113 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.437739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52052 (local address 127.2.74.253:43483)
0925 22:43:32.437858 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:32.437878 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:32.438180 (+ 302us) server_negotiation.cc:407] Connection header received
0925 22:43:32.438225 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:32.438228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:32.438280 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:32.438350 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:32.439258 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.440063 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.441159 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.441499 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.441603 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:32.444800 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:32.444807 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:32.444813 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:32.444847 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:32.446612 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:32.447214 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:32.447219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:32.447223 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:32.447314 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:32.447633 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:32.447637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:32.447638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:32.447883 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:32.448088 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:32.448822 (+ 734us) server_negotiation.cc:299] Negotiation successful
0925 22:43:32.448953 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 22:43:32.450448 15901 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3ba589dbaae049e2bcd69d876db86c64: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:43:32.450593 15901 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:43:32.452805 2347 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Sep 25 22:43:32 dist-test-slave-csnl krb5kdc[15636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376212, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.252@KRBTEST.COM for kudu/127.2.74.252@KRBTEST.COM
I20260925 22:43:32.455381 15878 raft_consensus.cc:482] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:43:32.455585 15878 raft_consensus.cc:504] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba589dbaae049e2bcd69d876db86c64" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36829 } } peers { permanent_uuid: "de70d0a3b83b4d0aa286fb606079b831" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 43483 } } peers { permanent_uuid: "d6feedda6f21464c94f23104a024808c" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 35083 } }
I20260925 22:43:32.451607 15649 master_runner.cc:186] Error getting master registration for 127.2.74.252:35083: OK, instance_id { permanent_uuid: "d6feedda6f21464c94f23104a024808c" instance_seqno: 1790376212399233 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:43:32.456712 15649 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:43:32.456703 15878 leader_election.cc:288] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ba589dbaae049e2bcd69d876db86c64 (127.2.74.254:36829), d6feedda6f21464c94f23104a024808c (127.2.74.252:35083)
I20260925 22:43:32.457176 15879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ba589dbaae049e2bcd69d876db86c64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba589dbaae049e2bcd69d876db86c64" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36829 } } peers { permanent_uuid: "de70d0a3b83b4d0aa286fb606079b831" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 43483 } } peers { permanent_uuid: "d6feedda6f21464c94f23104a024808c" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 35083 } } }
I20260925 22:43:32.457382 15879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ba589dbaae049e2bcd69d876db86c64 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:43:32.457939 15691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de70d0a3b83b4d0aa286fb606079b831" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba589dbaae049e2bcd69d876db86c64" is_pre_election: true
I20260925 22:43:32.458361 15691 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ba589dbaae049e2bcd69d876db86c64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de70d0a3b83b4d0aa286fb606079b831 in term 0.
I20260925 22:43:32.458734 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.445595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:35083 (local address 127.0.0.1:33018)
0925 22:43:32.445873 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:32.445887 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:32.445961 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:32.449118 (+ 3157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:32.449121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:32.449316 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:43:32.449420 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:32.449427 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.450441 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.450445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:32.451248 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:32.451258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.451894 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.451897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:32.451977 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:32.452708 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:43:32.452727 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:43:32.454408 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:43:32.456607 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:32.456610 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:32.456612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:32.456892 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:32.457445 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:32.457448 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:32.457450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:32.457545 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:32.458112 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:43:32.458115 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:43:32.458218 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 22:43:32.458336 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 22:43:32.458521 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193}
I20260925 22:43:32.458878 15743 leader_election.cc:302] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba589dbaae049e2bcd69d876db86c64, de70d0a3b83b4d0aa286fb606079b831; no voters:
I20260925 22:43:32.459329 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.445832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33018 (local address 127.2.74.252:35083)
0925 22:43:32.448825 (+ 2993us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:32.448830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:32.448845 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:43:32.448886 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:32.448889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:32.448936 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:32.449017 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:32.449564 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.450317 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.451372 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.451683 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.451780 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:32.454585 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:32.454589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:32.454592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:32.454618 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:32.456441 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:32.457185 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:32.457189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:32.457191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:32.457320 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:32.457673 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:32.457677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:32.457679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:32.457957 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:32.458071 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:32.459015 (+ 944us) server_negotiation.cc:299] Negotiation successful
0925 22:43:32.459151 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2892}
I20260925 22:43:32.459312 15878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:43:32.459412 15878 raft_consensus.cc:482] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:43:32.459472 15878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831 [term 0 FOLLOWER]: Advancing to term 1
W20260925 22:43:32.460630 15890 catalog_manager.cc:1558] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:43:32.460722 15890 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:43:32.461133 15878 raft_consensus.cc:504] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba589dbaae049e2bcd69d876db86c64" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36829 } } peers { permanent_uuid: "de70d0a3b83b4d0aa286fb606079b831" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 43483 } } peers { permanent_uuid: "d6feedda6f21464c94f23104a024808c" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 35083 } }
I20260925 22:43:32.461706 15878 leader_election.cc:288] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831 [CANDIDATE]: Term 1 election: Requested vote from peers 3ba589dbaae049e2bcd69d876db86c64 (127.2.74.254:36829), d6feedda6f21464c94f23104a024808c (127.2.74.252:35083)
I20260925 22:43:32.463577 15691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de70d0a3b83b4d0aa286fb606079b831" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba589dbaae049e2bcd69d876db86c64"
I20260925 22:43:32.463732 15691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ba589dbaae049e2bcd69d876db86c64 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:43:32.465123 15691 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ba589dbaae049e2bcd69d876db86c64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de70d0a3b83b4d0aa286fb606079b831 in term 1.
I20260925 22:43:32.465219 15868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6feedda6f21464c94f23104a024808c: Bootstrap starting.
I20260925 22:43:32.465708 15743 leader_election.cc:302] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba589dbaae049e2bcd69d876db86c64, de70d0a3b83b4d0aa286fb606079b831; no voters:
I20260925 22:43:32.467084 15878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:43:32.467690 15878 raft_consensus.cc:686] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831 [term 1 LEADER]: Becoming Leader. State: Replica: de70d0a3b83b4d0aa286fb606079b831, State: Running, Role: LEADER
I20260925 22:43:32.468393 15878 consensus_queue.cc:237] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba589dbaae049e2bcd69d876db86c64" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36829 } } peers { permanent_uuid: "de70d0a3b83b4d0aa286fb606079b831" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 43483 } } peers { permanent_uuid: "d6feedda6f21464c94f23104a024808c" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 35083 } }
I20260925 22:43:32.468794 15868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6feedda6f21464c94f23104a024808c: Neither blocks nor log segments found. Creating new log.
I20260925 22:43:32.469835 15868 log.cc:824] T 00000000000000000000000000000000 P d6feedda6f21464c94f23104a024808c: Log is configured to *not* fsync() on all Append() calls
I20260925 22:43:32.470516 15878 sys_catalog.cc:455] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de70d0a3b83b4d0aa286fb606079b831. Latest consensus state: current_term: 1 leader_uuid: "de70d0a3b83b4d0aa286fb606079b831" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba589dbaae049e2bcd69d876db86c64" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36829 } } peers { permanent_uuid: "de70d0a3b83b4d0aa286fb606079b831" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 43483 } } peers { permanent_uuid: "d6feedda6f21464c94f23104a024808c" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 35083 } } }
I20260925 22:43:32.470664 15878 sys_catalog.cc:458] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831 [sys.catalog]: This master's current role is: LEADER
I20260925 22:43:32.472263 15875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.458159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35510 (local address 127.2.74.254:36829)
0925 22:43:32.458278 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:32.458282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:32.460462 (+ 2180us) server_negotiation.cc:407] Connection header received
0925 22:43:32.460593 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:32.460596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:32.460657 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:32.460747 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:32.461874 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.462762 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.464162 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.464531 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.464623 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:32.466768 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:32.466772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:32.466775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:32.466808 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:32.468752 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:32.469755 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:32.469759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:32.469761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:32.469884 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:32.470506 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:32.470509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:32.470511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:32.470767 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:32.470894 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:32.471931 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0925 22:43:32.472091 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:43:32.472522 15868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6feedda6f21464c94f23104a024808c: No bootstrap required, opened a new log
I20260925 22:43:32.472904 15914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:43:32.474010 15914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:43:32.477852 15868 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6feedda6f21464c94f23104a024808c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba589dbaae049e2bcd69d876db86c64" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36829 } } peers { permanent_uuid: "de70d0a3b83b4d0aa286fb606079b831" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 43483 } } peers { permanent_uuid: "d6feedda6f21464c94f23104a024808c" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 35083 } }
I20260925 22:43:32.478228 15868 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6feedda6f21464c94f23104a024808c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:43:32.478313 15868 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6feedda6f21464c94f23104a024808c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6feedda6f21464c94f23104a024808c, State: Initialized, Role: FOLLOWER
I20260925 22:43:32.479064 15868 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6feedda6f21464c94f23104a024808c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba589dbaae049e2bcd69d876db86c64" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36829 } } peers { permanent_uuid: "de70d0a3b83b4d0aa286fb606079b831" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 43483 } } peers { permanent_uuid: "d6feedda6f21464c94f23104a024808c" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 35083 } }
I20260925 22:43:32.480927 15868 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6feedda6f21464c94f23104a024808c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:43:32.481205 15841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de70d0a3b83b4d0aa286fb606079b831" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6feedda6f21464c94f23104a024808c" is_pre_election: true
I20260925 22:43:32.481549 15841 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d6feedda6f21464c94f23104a024808c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de70d0a3b83b4d0aa286fb606079b831 in term 0.
I20260925 22:43:32.481833 15840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de70d0a3b83b4d0aa286fb606079b831" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6feedda6f21464c94f23104a024808c"
I20260925 22:43:32.481947 15840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6feedda6f21464c94f23104a024808c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:43:32.483502 15840 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d6feedda6f21464c94f23104a024808c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de70d0a3b83b4d0aa286fb606079b831 in term 1.
I20260925 22:43:32.485580 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.463217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52060 (local address 127.2.74.253:43483)
0925 22:43:32.466179 (+ 2962us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:32.466186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:32.467294 (+ 1108us) server_negotiation.cc:407] Connection header received
0925 22:43:32.467500 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:32.467506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:32.467572 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:32.467629 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:32.475057 (+ 7428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.475827 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.477155 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.477507 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.477641 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:32.479902 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:32.479907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:32.479910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:32.479946 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:32.481946 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:32.482609 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:32.482616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:32.482619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:32.482723 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:32.483175 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:32.483182 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:32.483188 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:32.483478 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:32.483617 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:32.485225 (+ 1608us) server_negotiation.cc:299] Negotiation successful
0925 22:43:32.485388 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2830}
I20260925 22:43:32.484287 15916 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6feedda6f21464c94f23104a024808c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba589dbaae049e2bcd69d876db86c64" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36829 } } peers { permanent_uuid: "de70d0a3b83b4d0aa286fb606079b831" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 43483 } } peers { permanent_uuid: "d6feedda6f21464c94f23104a024808c" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 35083 } } }
I20260925 22:43:32.488329 15916 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6feedda6f21464c94f23104a024808c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:43:32.489707 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:32.460007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33020 (local address 127.2.74.252:35083)
0925 22:43:32.460117 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:32.460121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:32.475415 (+ 15294us) server_negotiation.cc:407] Connection header received
0925 22:43:32.475694 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:32.475698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:32.475748 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:32.475797 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:32.477464 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.478208 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.480621 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:32.480970 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:32.481055 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:32.485666 (+ 4611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:32.485671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:32.485673 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:32.485704 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:32.487564 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:32.488074 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:32.488078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:32.488080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:32.488168 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:32.488647 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:32.488651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:32.488653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:32.488886 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:32.489032 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:32.489413 (+ 381us) server_negotiation.cc:299] Negotiation successful
0925 22:43:32.489544 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:43:32.493446 15840 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d6feedda6f21464c94f23104a024808c [term 1 FOLLOWER]: Refusing update from remote peer de70d0a3b83b4d0aa286fb606079b831: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:43:32.493670 15691 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ba589dbaae049e2bcd69d876db86c64 [term 1 FOLLOWER]: Refusing update from remote peer de70d0a3b83b4d0aa286fb606079b831: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:43:32.494161 15911 consensus_queue.cc:1035] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6feedda6f21464c94f23104a024808c" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 35083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:43:32.494382 15878 consensus_queue.cc:1035] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba589dbaae049e2bcd69d876db86c64" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:43:32.498659 15916 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6feedda6f21464c94f23104a024808c [sys.catalog]: SysCatalogTable state changed. Reason: New leader de70d0a3b83b4d0aa286fb606079b831. Latest consensus state: current_term: 1 leader_uuid: "de70d0a3b83b4d0aa286fb606079b831" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba589dbaae049e2bcd69d876db86c64" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36829 } } peers { permanent_uuid: "de70d0a3b83b4d0aa286fb606079b831" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 43483 } } peers { permanent_uuid: "d6feedda6f21464c94f23104a024808c" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 35083 } } }
I20260925 22:43:32.498816 15916 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6feedda6f21464c94f23104a024808c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:43:32.498764 15879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ba589dbaae049e2bcd69d876db86c64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de70d0a3b83b4d0aa286fb606079b831. Latest consensus state: current_term: 1 leader_uuid: "de70d0a3b83b4d0aa286fb606079b831" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba589dbaae049e2bcd69d876db86c64" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36829 } } peers { permanent_uuid: "de70d0a3b83b4d0aa286fb606079b831" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 43483 } } peers { permanent_uuid: "d6feedda6f21464c94f23104a024808c" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 35083 } } }
I20260925 22:43:32.498896 15879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ba589dbaae049e2bcd69d876db86c64 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 22:43:32.500461 15927 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d6feedda6f21464c94f23104a024808c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:43:32.500558 15927 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:43:32.503120 15911 sys_catalog.cc:455] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "de70d0a3b83b4d0aa286fb606079b831" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba589dbaae049e2bcd69d876db86c64" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36829 } } peers { permanent_uuid: "de70d0a3b83b4d0aa286fb606079b831" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 43483 } } peers { permanent_uuid: "d6feedda6f21464c94f23104a024808c" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 35083 } } }
I20260925 22:43:32.503324 15911 sys_catalog.cc:458] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831 [sys.catalog]: This master's current role is: LEADER
I20260925 22:43:32.504714 15879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ba589dbaae049e2bcd69d876db86c64 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "de70d0a3b83b4d0aa286fb606079b831" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba589dbaae049e2bcd69d876db86c64" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36829 } } peers { permanent_uuid: "de70d0a3b83b4d0aa286fb606079b831" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 43483 } } peers { permanent_uuid: "d6feedda6f21464c94f23104a024808c" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 35083 } } }
I20260925 22:43:32.504884 15879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ba589dbaae049e2bcd69d876db86c64 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:43:32.505455 15914 catalog_manager.cc:1347] Generated new cluster ID: 587221dc7133490db0844898803d1777
I20260925 22:43:32.505530 15914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:43:32.514361 15930 mvcc.cc:205] Tried to move back new op lower bound from 7333380966365593600 to 7333380966274551808. Current Snapshot: MvccSnapshot[applied={T|T < 7333380966365593600}]
I20260925 22:43:32.516876 15878 sys_catalog.cc:455] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "de70d0a3b83b4d0aa286fb606079b831" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba589dbaae049e2bcd69d876db86c64" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36829 } } peers { permanent_uuid: "de70d0a3b83b4d0aa286fb606079b831" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 43483 } } peers { permanent_uuid: "d6feedda6f21464c94f23104a024808c" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 35083 } } }
I20260925 22:43:32.517045 15878 sys_catalog.cc:458] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831 [sys.catalog]: This master's current role is: LEADER
I20260925 22:43:32.518675 15916 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6feedda6f21464c94f23104a024808c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "de70d0a3b83b4d0aa286fb606079b831" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba589dbaae049e2bcd69d876db86c64" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 36829 } } peers { permanent_uuid: "de70d0a3b83b4d0aa286fb606079b831" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 43483 } } peers { permanent_uuid: "d6feedda6f21464c94f23104a024808c" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 35083 } } }
I20260925 22:43:32.518860 15916 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6feedda6f21464c94f23104a024808c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:43:32.521888 15914 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:43:32.523505 15914 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:43:32.526789 15929 mvcc.cc:205] Tried to move back new op lower bound from 7333380966365593600 to 7333380966274551808. Current Snapshot: MvccSnapshot[applied={T|T < 7333380966365593600}]
I20260925 22:43:32.537541 15914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de70d0a3b83b4d0aa286fb606079b831: Generated new TSK 0
I20260925 22:43:32.538626 15914 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:43:33.452081 15901 catalog_manager.cc:1259] Loaded cluster ID: 587221dc7133490db0844898803d1777
I20260925 22:43:33.452215 15901 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3ba589dbaae049e2bcd69d876db86c64: loading cluster ID for follower catalog manager: success
I20260925 22:43:33.454231 15901 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3ba589dbaae049e2bcd69d876db86c64: acquiring CA information for follower catalog manager: success
I20260925 22:43:33.455691 15901 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3ba589dbaae049e2bcd69d876db86c64: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:43:33.502007 15927 catalog_manager.cc:1259] Loaded cluster ID: 587221dc7133490db0844898803d1777
I20260925 22:43:33.502110 15927 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d6feedda6f21464c94f23104a024808c: loading cluster ID for follower catalog manager: success
I20260925 22:43:33.504104 15927 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d6feedda6f21464c94f23104a024808c: acquiring CA information for follower catalog manager: success
I20260925 22:43:33.505576 15927 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d6feedda6f21464c94f23104a024808c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:43:33.606683 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 15649
I20260925 22:43:33.613667 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 15724
I20260925 22:43:33.620515 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 15799
2026-09-25T22:43:33Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3051 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:43:33 dist-test-slave-csnl krb5kdc[15942](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:43:33 dist-test-slave-csnl krb5kdc[15942](info): set up 2 sockets
Sep 25 22:43:33 dist-test-slave-csnl krb5kdc[15942](info): commencing operation
krb5kdc: starting...
W20260925 22:43:34.659349 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:43:34 dist-test-slave-csnl krb5kdc[15942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:43:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:43:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:43:34.786443 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:38121
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:36693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:38121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:43:34.906913 15955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:43:34.907285 15955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:43:34.907343 15955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:43:34.911558 15955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:43:34.911634 15955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:43:34.911654 15955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:43:34.911672 15955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:43:34.911688 15955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:43:34.917547 15955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:36693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:38121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:38121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:43:34.918695 15955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:43:34.919755 15955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:43:34.928948 15960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:34.928939 15961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:34.929083 15955 server_base.cc:1034] running on GCE node
W20260925 22:43:34.928939 15963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:34.929898 15955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:43:34.931210 15955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:43:34.932426 15955 hybrid_clock.cc:648] HybridClock initialized: now 1790376214932407 us; error 35 us; skew 500 ppm
Sep 25 22:43:34 dist-test-slave-csnl krb5kdc[15942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376214, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:43:34.935405 15955 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:43:34.936472 15955 webserver.cc:460] Webserver started at http://127.2.74.254:41057/ using document root <none> and password file <none>
I20260925 22:43:34.937095 15955 fs_manager.cc:359] Metadata directory not provided
I20260925 22:43:34.937197 15955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:43:34.937453 15955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:43:34.939744 15955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "cf3c363455924a9db5021b96c77bf526"
format_stamp: "Formatted at 2026-09-25 22:43:34 on dist-test-slave-csnl"
server_key: "f7acf4177445b3068366c6ba7355fa67"
server_key_iv: "5c3974be4230322a503249e17d1bec35"
server_key_version: "encryptionkey@0"
I20260925 22:43:34.940361 15955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "cf3c363455924a9db5021b96c77bf526"
format_stamp: "Formatted at 2026-09-25 22:43:34 on dist-test-slave-csnl"
server_key: "f7acf4177445b3068366c6ba7355fa67"
server_key_iv: "5c3974be4230322a503249e17d1bec35"
server_key_version: "encryptionkey@0"
I20260925 22:43:34.944621 15955 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:43:34.947302 15970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:34.948639 15955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:43:34.948803 15955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "cf3c363455924a9db5021b96c77bf526"
format_stamp: "Formatted at 2026-09-25 22:43:34 on dist-test-slave-csnl"
server_key: "f7acf4177445b3068366c6ba7355fa67"
server_key_iv: "5c3974be4230322a503249e17d1bec35"
server_key_version: "encryptionkey@0"
I20260925 22:43:34.948951 15955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:43:34.961207 15955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:43:34.964802 15955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:43:34.965029 15955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:43:34.973285 15955 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:38121
I20260925 22:43:34.973456 16022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:38121 every 8 connection(s)
I20260925 22:43:34.974367 15955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:43:34.977372 16023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:43:34.983579 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 15955
I20260925 22:43:34.983723 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:43:34.983801 16023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf3c363455924a9db5021b96c77bf526: Bootstrap starting.
I20260925 22:43:34.984081 2347 external_mini_cluster.cc:1428] Setting key dd86de3d5e6f992ca94cec90597fd04d
I20260925 22:43:34.986657 16023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf3c363455924a9db5021b96c77bf526: Neither blocks nor log segments found. Creating new log.
I20260925 22:43:34.987619 16023 log.cc:824] T 00000000000000000000000000000000 P cf3c363455924a9db5021b96c77bf526: Log is configured to *not* fsync() on all Append() calls
I20260925 22:43:34.989884 16023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf3c363455924a9db5021b96c77bf526: No bootstrap required, opened a new log
Sep 25 22:43:34 dist-test-slave-csnl krb5kdc[15942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:43:34.994781 16023 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf3c363455924a9db5021b96c77bf526 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3c363455924a9db5021b96c77bf526" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38121 } }
I20260925 22:43:34.995249 16023 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf3c363455924a9db5021b96c77bf526 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:43:34.995328 16023 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf3c363455924a9db5021b96c77bf526 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf3c363455924a9db5021b96c77bf526, State: Initialized, Role: FOLLOWER
I20260925 22:43:34.995918 16023 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf3c363455924a9db5021b96c77bf526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3c363455924a9db5021b96c77bf526" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38121 } }
I20260925 22:43:34.996109 16023 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf3c363455924a9db5021b96c77bf526 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:43:34.996214 16023 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf3c363455924a9db5021b96c77bf526 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:43:34.996348 16023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf3c363455924a9db5021b96c77bf526 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:43:34.997370 16023 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf3c363455924a9db5021b96c77bf526 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3c363455924a9db5021b96c77bf526" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38121 } }
I20260925 22:43:34.997764 16023 leader_election.cc:302] T 00000000000000000000000000000000 P cf3c363455924a9db5021b96c77bf526 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf3c363455924a9db5021b96c77bf526; no voters:
I20260925 22:43:34.998178 16023 leader_election.cc:288] T 00000000000000000000000000000000 P cf3c363455924a9db5021b96c77bf526 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:43:34.998231 16028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf3c363455924a9db5021b96c77bf526 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:43:34.998464 16028 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf3c363455924a9db5021b96c77bf526 [term 1 LEADER]: Becoming Leader. State: Replica: cf3c363455924a9db5021b96c77bf526, State: Running, Role: LEADER
I20260925 22:43:34.998826 16028 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf3c363455924a9db5021b96c77bf526 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3c363455924a9db5021b96c77bf526" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38121 } }
I20260925 22:43:35.000207 16028 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf3c363455924a9db5021b96c77bf526 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf3c363455924a9db5021b96c77bf526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3c363455924a9db5021b96c77bf526" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38121 } } }
I20260925 22:43:35.000348 16028 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf3c363455924a9db5021b96c77bf526 [sys.catalog]: This master's current role is: LEADER
I20260925 22:43:35.000411 16023 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf3c363455924a9db5021b96c77bf526 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:43:35.000633 16029 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf3c363455924a9db5021b96c77bf526 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf3c363455924a9db5021b96c77bf526. Latest consensus state: current_term: 1 leader_uuid: "cf3c363455924a9db5021b96c77bf526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3c363455924a9db5021b96c77bf526" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 38121 } } }
I20260925 22:43:35.000725 16029 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf3c363455924a9db5021b96c77bf526 [sys.catalog]: This master's current role is: LEADER
I20260925 22:43:35.000726 16032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:43:35.001170 16026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:34.985616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45638 (local address 127.2.74.254:38121)
0925 22:43:34.986075 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:34.986092 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:34.986131 (+ 39us) server_negotiation.cc:407] Connection header received
0925 22:43:34.986917 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:34.986940 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:34.987376 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:34.987698 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:34.988677 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:34.989693 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:34.990897 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:34.991474 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:34.991618 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:34.994684 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:34.994691 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:34.994707 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:34.994742 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:34.996963 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:34.997496 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:34.997502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:34.997510 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:34.997653 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:34.997969 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:34.997973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:34.997974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:34.998611 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:34.999509 (+ 898us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:34.999701 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 22:43:34.999980 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":157,"threads_started":1}
I20260925 22:43:35.004259 16032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:43:35.004473 2347 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:43:35.010492 16032 catalog_manager.cc:1347] Generated new cluster ID: dde5ba389d10429d89a0b9611ebb1efc
I20260925 22:43:35.010628 16032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:43:35.020203 16032 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:43:35.021330 16032 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:43:35.027366 16032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf3c363455924a9db5021b96c77bf526: Generated new TSK 0
I20260925 22:43:35.028188 16032 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:43:35.061210 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 15955
2026-09-25T22:43:35Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1438 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:43:35 dist-test-slave-csnl krb5kdc[16050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:43:35 dist-test-slave-csnl krb5kdc[16050](info): set up 2 sockets
Sep 25 22:43:35 dist-test-slave-csnl krb5kdc[16050](info): commencing operation
krb5kdc: starting...
W20260925 22:43:36.103484 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:43:36 dist-test-slave-csnl krb5kdc[16050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:43:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:43:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:43:36.231611 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:34089
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:34089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:43:36.345906 16063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:43:36.346248 16063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:43:36.346294 16063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:43:36.350612 16063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:43:36.350685 16063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:43:36.350705 16063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:43:36.350723 16063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:43:36.350739 16063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:43:36.356386 16063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:34089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:34089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:43:36.357566 16063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:43:36.358690 16063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:43:36.367561 16071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:36.367566 16068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:36.367614 16063 server_base.cc:1034] running on GCE node
W20260925 22:43:36.367894 16069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:36.368500 16063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:43:36.369735 16063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:43:36.371009 16063 hybrid_clock.cc:648] HybridClock initialized: now 1790376216370915 us; error 48 us; skew 500 ppm
Sep 25 22:43:36 dist-test-slave-csnl krb5kdc[16050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376216, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:43:36.379895 16063 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:43:36.380975 16063 webserver.cc:460] Webserver started at http://127.2.74.254:44589/ using document root <none> and password file <none>
I20260925 22:43:36.381629 16063 fs_manager.cc:359] Metadata directory not provided
I20260925 22:43:36.381706 16063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:43:36.381904 16063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:43:36.384262 16063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "48ab6537b643474689184dfc76cba2af"
format_stamp: "Formatted at 2026-09-25 22:43:36 on dist-test-slave-csnl"
server_key: "279b47f6bc8d968eb07c9149acb6336f"
server_key_iv: "8b8674ac339906089d84ddbc4e68fcb9"
server_key_version: "encryptionkey@0"
I20260925 22:43:36.384845 16063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "48ab6537b643474689184dfc76cba2af"
format_stamp: "Formatted at 2026-09-25 22:43:36 on dist-test-slave-csnl"
server_key: "279b47f6bc8d968eb07c9149acb6336f"
server_key_iv: "8b8674ac339906089d84ddbc4e68fcb9"
server_key_version: "encryptionkey@0"
I20260925 22:43:36.388994 16063 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:43:36.391822 16078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:36.393288 16063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:43:36.393460 16063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "48ab6537b643474689184dfc76cba2af"
format_stamp: "Formatted at 2026-09-25 22:43:36 on dist-test-slave-csnl"
server_key: "279b47f6bc8d968eb07c9149acb6336f"
server_key_iv: "8b8674ac339906089d84ddbc4e68fcb9"
server_key_version: "encryptionkey@0"
I20260925 22:43:36.393589 16063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:43:36.408833 16063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:43:36.412122 16063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:43:36.412295 16063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:43:36.419792 16063 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:34089
I20260925 22:43:36.420132 16130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:34089 every 8 connection(s)
I20260925 22:43:36.420897 16063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:43:36.424196 16131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:43:36.428679 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 16063
I20260925 22:43:36.428845 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:43:36.429152 2347 external_mini_cluster.cc:1428] Setting key 0db16ddc96a7bca49a56bb63869c1945
I20260925 22:43:36.431160 16131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af: Bootstrap starting.
I20260925 22:43:36.433813 16131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af: Neither blocks nor log segments found. Creating new log.
I20260925 22:43:36.434623 16131 log.cc:824] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af: Log is configured to *not* fsync() on all Append() calls
I20260925 22:43:36.437060 16131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af: No bootstrap required, opened a new log
Sep 25 22:43:36 dist-test-slave-csnl krb5kdc[16050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:43:36.441675 16131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ab6537b643474689184dfc76cba2af" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34089 } }
I20260925 22:43:36.442003 16131 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:43:36.442060 16131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48ab6537b643474689184dfc76cba2af, State: Initialized, Role: FOLLOWER
I20260925 22:43:36.442581 16131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ab6537b643474689184dfc76cba2af" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34089 } }
I20260925 22:43:36.442747 16131 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:43:36.442822 16131 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:43:36.442963 16131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:43:36.444280 16131 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ab6537b643474689184dfc76cba2af" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34089 } }
I20260925 22:43:36.444669 16131 leader_election.cc:302] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48ab6537b643474689184dfc76cba2af; no voters:
I20260925 22:43:36.445025 16131 leader_election.cc:288] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:43:36.445149 16136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:43:36.445436 16136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [term 1 LEADER]: Becoming Leader. State: Replica: 48ab6537b643474689184dfc76cba2af, State: Running, Role: LEADER
I20260925 22:43:36.445712 16134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:36.430717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34060 (local address 127.2.74.254:34089)
0925 22:43:36.431185 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:36.431202 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:36.431238 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:43:36.431984 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:36.432002 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:36.432366 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:36.432665 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:36.433605 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:36.434715 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:36.435832 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:36.436304 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:36.436462 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:36.439395 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:36.439402 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:36.439422 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:36.439458 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:36.442193 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:36.442802 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:36.442809 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:36.442815 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:36.442994 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:36.443319 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:36.443322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:36.443325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:36.443876 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:36.444089 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:36.444373 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 22:43:36.444691 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":173,"threads_started":1}
I20260925 22:43:36.445801 16136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ab6537b643474689184dfc76cba2af" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34089 } }
I20260925 22:43:36.446302 16131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:43:36.447623 16137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48ab6537b643474689184dfc76cba2af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ab6537b643474689184dfc76cba2af" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34089 } } }
I20260925 22:43:36.447743 16137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [sys.catalog]: This master's current role is: LEADER
I20260925 22:43:36.448093 16138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48ab6537b643474689184dfc76cba2af. Latest consensus state: current_term: 1 leader_uuid: "48ab6537b643474689184dfc76cba2af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ab6537b643474689184dfc76cba2af" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34089 } } }
I20260925 22:43:36.448233 16138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [sys.catalog]: This master's current role is: LEADER
I20260925 22:43:36.448550 16145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:43:36.451360 16145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:43:36.457676 16145 catalog_manager.cc:1347] Generated new cluster ID: fb6bdf8d2fbe47b49e4f57a10047c990
I20260925 22:43:36.457763 16145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:43:36.485071 16145 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:43:36.486433 16145 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:43:36.495249 16145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af: Generated new TSK 0
I20260925 22:43:36.496028 16145 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
I20260925 22:43:36.534039 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:0
--local_ip_for_outbound_sockets=127.2.74.193
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:34089
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:43:36.653369 16157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:43:36.653674 16157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:43:36.653748 16157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:43:36.658082 16157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:43:36.658170 16157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:43:36.658274 16157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:43:36.664361 16157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:34089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:43:36.665633 16157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:43:36.666775 16157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:43:36.677251 16165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:36.677301 16163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:36.677260 16162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:36.677492 16157 server_base.cc:1034] running on GCE node
I20260925 22:43:36.678112 16157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:43:36.678819 16157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:43:36.679987 16157 hybrid_clock.cc:648] HybridClock initialized: now 1790376216679968 us; error 35 us; skew 500 ppm
Sep 25 22:43:36 dist-test-slave-csnl krb5kdc[16050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376216, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:43:36.682924 16157 init.cc:377] Logged in from keytab as kudu/127.2.74.193@KRBTEST.COM (short username kudu)
I20260925 22:43:36.683969 16157 webserver.cc:460] Webserver started at http://127.2.74.193:46723/ using document root <none> and password file <none>
I20260925 22:43:36.684594 16157 fs_manager.cc:359] Metadata directory not provided
I20260925 22:43:36.684695 16157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:43:36.684933 16157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:43:36.687186 16157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/data/instance:
uuid: "09608869c3374b659697b7b552d79c0a"
format_stamp: "Formatted at 2026-09-25 22:43:36 on dist-test-slave-csnl"
server_key: "06158e1ff88d038207b57438cbc1fa8d"
server_key_iv: "23cd8e23caaacc4f78f250f7cf905c35"
server_key_version: "encryptionkey@0"
I20260925 22:43:36.687780 16157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance:
uuid: "09608869c3374b659697b7b552d79c0a"
format_stamp: "Formatted at 2026-09-25 22:43:36 on dist-test-slave-csnl"
server_key: "06158e1ff88d038207b57438cbc1fa8d"
server_key_iv: "23cd8e23caaacc4f78f250f7cf905c35"
server_key_version: "encryptionkey@0"
I20260925 22:43:36.691864 16157 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:43:36.694525 16172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:36.695967 16157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:43:36.696141 16157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "09608869c3374b659697b7b552d79c0a"
format_stamp: "Formatted at 2026-09-25 22:43:36 on dist-test-slave-csnl"
server_key: "06158e1ff88d038207b57438cbc1fa8d"
server_key_iv: "23cd8e23caaacc4f78f250f7cf905c35"
server_key_version: "encryptionkey@0"
I20260925 22:43:36.696273 16157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:43:36.714299 16157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:43:36.717422 16157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:43:36.717651 16157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:43:36.718269 16157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:43:36.719197 16157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:43:36.719267 16157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:36.719358 16157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:43:36.719404 16157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:36.728545 16157 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:41583
I20260925 22:43:36.728576 16285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:41583 every 8 connection(s)
I20260925 22:43:36.729486 16157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:43:36.731381 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 16157
I20260925 22:43:36.731539 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance
I20260925 22:43:36.731844 2347 external_mini_cluster.cc:1428] Setting key 2c3fa435d2a729a82d9f5e12e1ebd0a7
Sep 25 22:43:36 dist-test-slave-csnl krb5kdc[16050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376216, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.74.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.194@KRBTEST.COM" created.
I20260925 22:43:36.745493 16134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:36.731563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:46211 (local address 127.2.74.254:34089)
0925 22:43:36.731710 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:36.731715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:36.732810 (+ 1095us) server_negotiation.cc:407] Connection header received
0925 22:43:36.734198 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:36.734205 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:36.734273 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:36.734371 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:36.735772 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:36.736520 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:36.737960 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:36.738435 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:36.738581 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:36.741425 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:36.741433 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:36.741438 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:36.741479 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:36.743128 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:36.743797 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:36.743800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:36.743801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:36.743881 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:36.744320 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:36.744322 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:36.744323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:36.744519 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:36.744627 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:36.745187 (+ 560us) server_negotiation.cc:299] Negotiation successful
0925 22:43:36.745322 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 22:43:36.746733 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:36.732089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.193:46211)
0925 22:43:36.732631 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:36.732696 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:36.733943 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:36.734574 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:36.734582 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:36.735062 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:43:36.735601 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:36.735616 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:36.736702 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:36.736705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:36.737801 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:36.737814 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:36.738651 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:36.738654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:36.738841 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:36.739608 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:43:36.739628 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:43:36.741190 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:43:36.743266 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:36.743272 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:36.743289 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:36.743696 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:36.744012 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:36.744015 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:36.744017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:36.744231 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:36.744644 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:43:36.744653 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:43:36.744934 (+ 281us) client_negotiation.cc:769] Sending connection context
0925 22:43:36.745135 (+ 201us) client_negotiation.cc:240] Negotiation successful
0925 22:43:36.745497 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":144,"threads_started":1}
I20260925 22:43:36.748139 16286 heartbeater.cc:347] Connected to a master server at 127.2.74.254:34089
I20260925 22:43:36.748451 16286 heartbeater.cc:464] Registering TS with master...
I20260925 22:43:36.749027 16286 heartbeater.cc:511] Master 127.2.74.254:34089 requested a full tablet report, sending...
I20260925 22:43:36.750579 16095 ts_manager.cc:194] Registered new tserver with Master: 09608869c3374b659697b7b552d79c0a (127.2.74.193:41583)
I20260925 22:43:36.751801 16095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.193@KRBTEST.COM'} at 127.2.74.193:46211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
I20260925 22:43:36.763331 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:0
--local_ip_for_outbound_sockets=127.2.74.194
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:34089
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:43:36.881809 16291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:43:36.882159 16291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:43:36.882237 16291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:43:36.886576 16291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:43:36.886701 16291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:43:36.886803 16291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:43:36.892470 16291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:34089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:43:36.893631 16291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:43:36.894714 16291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:43:36.904889 16299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:36.904961 16291 server_base.cc:1034] running on GCE node
W20260925 22:43:36.904882 16296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:36.904889 16297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:36.905615 16291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:43:36.906347 16291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:43:36.907557 16291 hybrid_clock.cc:648] HybridClock initialized: now 1790376216907538 us; error 35 us; skew 500 ppm
Sep 25 22:43:36 dist-test-slave-csnl krb5kdc[16050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376216, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:43:36.910550 16291 init.cc:377] Logged in from keytab as kudu/127.2.74.194@KRBTEST.COM (short username kudu)
I20260925 22:43:36.911586 16291 webserver.cc:460] Webserver started at http://127.2.74.194:43337/ using document root <none> and password file <none>
I20260925 22:43:36.912135 16291 fs_manager.cc:359] Metadata directory not provided
I20260925 22:43:36.912201 16291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:43:36.912374 16291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:43:36.914528 16291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/data/instance:
uuid: "e0bc0bded1b14a7589d138d4bee9474b"
format_stamp: "Formatted at 2026-09-25 22:43:36 on dist-test-slave-csnl"
server_key: "e72ef7651d2168a7f8b05e9c1cd879b4"
server_key_iv: "37282bdc1b17f279721d3597822be6fa"
server_key_version: "encryptionkey@0"
I20260925 22:43:36.915166 16291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance:
uuid: "e0bc0bded1b14a7589d138d4bee9474b"
format_stamp: "Formatted at 2026-09-25 22:43:36 on dist-test-slave-csnl"
server_key: "e72ef7651d2168a7f8b05e9c1cd879b4"
server_key_iv: "37282bdc1b17f279721d3597822be6fa"
server_key_version: "encryptionkey@0"
I20260925 22:43:36.919210 16291 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:43:36.921832 16306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:36.923365 16291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:43:36.923511 16291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "e0bc0bded1b14a7589d138d4bee9474b"
format_stamp: "Formatted at 2026-09-25 22:43:36 on dist-test-slave-csnl"
server_key: "e72ef7651d2168a7f8b05e9c1cd879b4"
server_key_iv: "37282bdc1b17f279721d3597822be6fa"
server_key_version: "encryptionkey@0"
I20260925 22:43:36.923622 16291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:43:36.942183 16291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:43:36.945688 16291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:43:36.945969 16291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:43:36.946599 16291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:43:36.947628 16291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:43:36.947682 16291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:36.947757 16291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:43:36.947790 16291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:36.957404 16291 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:39825
I20260925 22:43:36.957417 16419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:39825 every 8 connection(s)
I20260925 22:43:36.958407 16291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:43:36.960903 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 16291
I20260925 22:43:36.961045 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance
I20260925 22:43:36.961304 2347 external_mini_cluster.cc:1428] Setting key cd04dd4f370b428dd29a74b636f2539e
Sep 25 22:43:36 dist-test-slave-csnl krb5kdc[16050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376216, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.74.195@KRBTEST.COM; defaulting to no policy
I20260925 22:43:36.974335 16134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:36.960423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:33423 (local address 127.2.74.254:34089)
0925 22:43:36.960576 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:36.960580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:36.961617 (+ 1037us) server_negotiation.cc:407] Connection header received
0925 22:43:36.963065 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:36.963069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:36.963128 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:36.963218 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:36.964604 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:36.965487 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:36.966673 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:36.967013 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:36.967113 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:36.970398 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:36.970403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:36.970406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:36.970443 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:36.972172 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:36.972727 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:36.972735 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:36.972739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:36.972836 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:36.973206 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:36.973209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:36.973210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:36.973411 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:36.973493 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:36.974074 (+ 581us) server_negotiation.cc:299] Negotiation successful
0925 22:43:36.974205 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.195@KRBTEST.COM" created.
I20260925 22:43:36.975449 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:36.960856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.194:33423)
0925 22:43:36.961431 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:36.961491 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:36.962752 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:36.963417 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:36.963425 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:36.963835 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:43:36.964415 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:36.964439 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:36.965636 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:36.965639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:36.966556 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:36.966566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:36.967181 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:36.967185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:36.967369 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:36.968106 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:43:36.968136 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:43:36.970169 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:43:36.972310 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:36.972315 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:36.972331 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:36.972610 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:36.972948 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:36.972951 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:36.972952 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:36.973112 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:36.973499 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:43:36.973504 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:43:36.973813 (+ 309us) client_negotiation.cc:769] Sending connection context
0925 22:43:36.974033 (+ 220us) client_negotiation.cc:240] Negotiation successful
0925 22:43:36.974406 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":180,"threads_started":1}
I20260925 22:43:36.976686 16420 heartbeater.cc:347] Connected to a master server at 127.2.74.254:34089
I20260925 22:43:36.976986 16420 heartbeater.cc:464] Registering TS with master...
I20260925 22:43:36.977597 16420 heartbeater.cc:511] Master 127.2.74.254:34089 requested a full tablet report, sending...
I20260925 22:43:36.978936 16095 ts_manager.cc:194] Registered new tserver with Master: e0bc0bded1b14a7589d138d4bee9474b (127.2.74.194:39825)
I20260925 22:43:36.979844 16095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.194@KRBTEST.COM'} at 127.2.74.194:33423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
I20260925 22:43:36.994596 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:0
--local_ip_for_outbound_sockets=127.2.74.195
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:34089
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:43:37.110401 16425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:43:37.110697 16425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:43:37.110742 16425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:43:37.114799 16425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:43:37.114879 16425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:43:37.114943 16425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:43:37.120538 16425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:34089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:43:37.121671 16425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:43:37.122675 16425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:43:37.132753 16433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:37.132812 16425 server_base.cc:1034] running on GCE node
W20260925 22:43:37.132791 16430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:37.132786 16431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:37.133466 16425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:43:37.134150 16425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:43:37.135344 16425 hybrid_clock.cc:648] HybridClock initialized: now 1790376217135320 us; error 32 us; skew 500 ppm
Sep 25 22:43:37 dist-test-slave-csnl krb5kdc[16050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376217, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:43:37.138145 16425 init.cc:377] Logged in from keytab as kudu/127.2.74.195@KRBTEST.COM (short username kudu)
I20260925 22:43:37.139182 16425 webserver.cc:460] Webserver started at http://127.2.74.195:37089/ using document root <none> and password file <none>
I20260925 22:43:37.139792 16425 fs_manager.cc:359] Metadata directory not provided
I20260925 22:43:37.139897 16425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:43:37.140131 16425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:43:37.142266 16425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/data/instance:
uuid: "ac3193b1872a438f822322b5164b036b"
format_stamp: "Formatted at 2026-09-25 22:43:37 on dist-test-slave-csnl"
server_key: "9dcfab03b37f82a8b3d27e9bf7f378c3"
server_key_iv: "636c8c5bab73c4b9d4581737f2e5abf8"
server_key_version: "encryptionkey@0"
I20260925 22:43:37.142884 16425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance:
uuid: "ac3193b1872a438f822322b5164b036b"
format_stamp: "Formatted at 2026-09-25 22:43:37 on dist-test-slave-csnl"
server_key: "9dcfab03b37f82a8b3d27e9bf7f378c3"
server_key_iv: "636c8c5bab73c4b9d4581737f2e5abf8"
server_key_version: "encryptionkey@0"
I20260925 22:43:37.146893 16425 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:43:37.149821 16440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:37.151286 16425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:43:37.151450 16425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "ac3193b1872a438f822322b5164b036b"
format_stamp: "Formatted at 2026-09-25 22:43:37 on dist-test-slave-csnl"
server_key: "9dcfab03b37f82a8b3d27e9bf7f378c3"
server_key_iv: "636c8c5bab73c4b9d4581737f2e5abf8"
server_key_version: "encryptionkey@0"
I20260925 22:43:37.151583 16425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:43:37.163887 16425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:43:37.167420 16425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:43:37.167667 16425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:43:37.168341 16425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:43:37.169296 16425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:43:37.169376 16425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:37.169456 16425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:43:37.169509 16425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:37.179031 16425 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:35695
I20260925 22:43:37.179067 16553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:35695 every 8 connection(s)
I20260925 22:43:37.180070 16425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:43:37.181526 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 16425
I20260925 22:43:37.181690 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance
I20260925 22:43:37.182034 2347 external_mini_cluster.cc:1428] Setting key b7e581299955a88299f854b1ddd952e9
Sep 25 22:43:37 dist-test-slave-csnl krb5kdc[16050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376217, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:43:37.195246 16134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:37.182227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:54467 (local address 127.2.74.254:34089)
0925 22:43:37.182387 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:37.182392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:37.183584 (+ 1192us) server_negotiation.cc:407] Connection header received
0925 22:43:37.185064 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:37.185067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:37.185125 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:37.185201 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:37.186466 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.187116 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.188369 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.188610 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.188730 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:37.191434 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:37.191443 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:37.191445 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:37.191471 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:37.192842 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:37.193439 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:37.193442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:37.193444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:37.193541 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:37.193915 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:37.193921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:37.193926 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:37.194134 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:37.194246 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:37.194891 (+ 645us) server_negotiation.cc:299] Negotiation successful
0925 22:43:37.195083 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:43:37.196228 16556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:37.182790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.195:54467)
0925 22:43:37.183433 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:37.183498 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:37.184844 (+ 1346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:37.185394 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:37.185401 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:37.185789 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:43:37.186289 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:37.186303 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.187251 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.187255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:37.188207 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:37.188217 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.188781 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.188785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:37.188972 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:37.189544 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:43:37.189562 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:43:37.191225 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:43:37.193008 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:37.193014 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:37.193039 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:37.193314 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:37.193641 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:37.193645 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:37.193647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:37.193800 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:37.194244 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:43:37.194251 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:43:37.194566 (+ 315us) client_negotiation.cc:769] Sending connection context
0925 22:43:37.194795 (+ 229us) client_negotiation.cc:240] Negotiation successful
0925 22:43:37.195195 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":159,"threads_started":1}
I20260925 22:43:37.197418 16554 heartbeater.cc:347] Connected to a master server at 127.2.74.254:34089
I20260925 22:43:37.197712 16554 heartbeater.cc:464] Registering TS with master...
I20260925 22:43:37.198287 16554 heartbeater.cc:511] Master 127.2.74.254:34089 requested a full tablet report, sending...
I20260925 22:43:37.199450 16095 ts_manager.cc:194] Registered new tserver with Master: ac3193b1872a438f822322b5164b036b (127.2.74.195:35695)
I20260925 22:43:37.200060 16095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.195@KRBTEST.COM'} at 127.2.74.195:54467
I20260925 22:43:37.208572 2347 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:43:37.218472 16134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:37.210651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34066 (local address 127.2.74.254:34089)
0925 22:43:37.210806 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:37.210811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:37.211039 (+ 228us) server_negotiation.cc:407] Connection header received
0925 22:43:37.211162 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:37.211164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:37.211211 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:37.211293 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:37.212008 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.212554 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.213419 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.213651 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.213740 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:37.215283 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:37.215286 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:37.215288 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:37.215313 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:37.216782 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:37.217226 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:37.217229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:37.217230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:37.217310 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:37.217550 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:37.217552 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:37.217553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:37.217796 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:37.217904 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:37.218112 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 22:43:37.218221 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:43:37.221812 16095 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:43:37.225111 16095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:43:37.241111 16568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:37.234743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:39825 (local address 127.0.0.1:41532)
0925 22:43:37.235703 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:37.235764 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:37.235966 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:37.236752 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:37.236758 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:37.236781 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:43:37.236959 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:37.236968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.238134 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.238140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:37.239547 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:37.239558 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.240677 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.240680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:37.240781 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:37.240803 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:43:37.240878 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 22:43:37.240946 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"thread_start_us":74,"threads_started":1}
I20260925 22:43:37.241263 16569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:37.235650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43968 (local address 127.2.74.195:35695)
0925 22:43:37.236501 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:37.236513 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:37.236534 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:43:37.236610 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:37.236615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:37.236805 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:37.236960 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:43:37.237177 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.238299 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.239717 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.240313 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.240427 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:37.240597 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:37.241027 (+ 430us) server_negotiation.cc:299] Negotiation successful
0925 22:43:37.241114 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":163,"threads_started":1}
I20260925 22:43:37.241168 16567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:37.234891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41532 (local address 127.2.74.194:39825)
0925 22:43:37.235439 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:37.235453 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:37.235866 (+ 413us) server_negotiation.cc:407] Connection header received
0925 22:43:37.236131 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:37.236138 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:37.236371 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:37.236513 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:43:37.237088 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.238014 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.239691 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.240548 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.240648 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:37.240797 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:37.240923 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 22:43:37.241025 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":170,"threads_started":1}
I20260925 22:43:37.241515 16565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:37.235157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:35695 (local address 127.0.0.1:43968)
0925 22:43:37.236121 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:37.236135 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:37.236213 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:37.236943 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:37.236945 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:37.236961 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:43:37.237076 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:37.237082 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.238439 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.238442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:37.239566 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:37.239572 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.240441 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.240444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:37.240556 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:37.240615 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 22:43:37.240718 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 22:43:37.240857 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":911,"thread_start_us":89,"threads_started":1}
I20260925 22:43:37.242172 16566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:37.234857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:41583 (local address 127.0.0.1:38120)
0925 22:43:37.235703 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:37.235763 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:37.235889 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:37.237698 (+ 1809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:37.237701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:37.237719 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:43:37.237806 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:37.237812 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.239164 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.239170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:37.240217 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:37.240228 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.241772 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.241776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:37.241876 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:37.241910 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 22:43:37.241985 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 22:43:37.242050 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"thread_start_us":158,"threads_started":1}
I20260925 22:43:37.242616 16570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:37.236081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38120 (local address 127.2.74.193:41583)
0925 22:43:37.237237 (+ 1156us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:37.237251 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:37.237272 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:43:37.237342 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:37.237347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:37.237541 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:37.237691 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:43:37.237935 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.239011 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.240359 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.241485 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.241744 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:37.241884 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:37.242337 (+ 453us) server_negotiation.cc:299] Negotiation successful
0925 22:43:37.242462 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1050,"thread_start_us":86,"threads_started":1}
I20260925 22:43:37.244948 16220 tablet_service.cc:1467] Processing CreateTablet for tablet b999e5930df74d2787b7564c0e6044e2 (DEFAULT_TABLE table=test-table [id=d5800fa89f554c02abac5b96af45d1c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:43:37.244800 16488 tablet_service.cc:1467] Processing CreateTablet for tablet b999e5930df74d2787b7564c0e6044e2 (DEFAULT_TABLE table=test-table [id=d5800fa89f554c02abac5b96af45d1c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:43:37.244800 16354 tablet_service.cc:1467] Processing CreateTablet for tablet b999e5930df74d2787b7564c0e6044e2 (DEFAULT_TABLE table=test-table [id=d5800fa89f554c02abac5b96af45d1c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:43:37.245952 16488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b999e5930df74d2787b7564c0e6044e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:43:37.245945 16354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b999e5930df74d2787b7564c0e6044e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:43:37.245983 16220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b999e5930df74d2787b7564c0e6044e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:43:37.252846 16571 tablet_bootstrap.cc:492] T b999e5930df74d2787b7564c0e6044e2 P 09608869c3374b659697b7b552d79c0a: Bootstrap starting.
I20260925 22:43:37.253931 16573 tablet_bootstrap.cc:492] T b999e5930df74d2787b7564c0e6044e2 P e0bc0bded1b14a7589d138d4bee9474b: Bootstrap starting.
I20260925 22:43:37.255338 16572 tablet_bootstrap.cc:492] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b: Bootstrap starting.
I20260925 22:43:37.256019 16571 tablet_bootstrap.cc:654] T b999e5930df74d2787b7564c0e6044e2 P 09608869c3374b659697b7b552d79c0a: Neither blocks nor log segments found. Creating new log.
I20260925 22:43:37.256446 16573 tablet_bootstrap.cc:654] T b999e5930df74d2787b7564c0e6044e2 P e0bc0bded1b14a7589d138d4bee9474b: Neither blocks nor log segments found. Creating new log.
I20260925 22:43:37.257028 16571 log.cc:824] T b999e5930df74d2787b7564c0e6044e2 P 09608869c3374b659697b7b552d79c0a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:43:37.257292 16573 log.cc:824] T b999e5930df74d2787b7564c0e6044e2 P e0bc0bded1b14a7589d138d4bee9474b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:43:37.258129 16572 tablet_bootstrap.cc:654] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b: Neither blocks nor log segments found. Creating new log.
I20260925 22:43:37.259053 16572 log.cc:824] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:43:37.259654 16571 tablet_bootstrap.cc:492] T b999e5930df74d2787b7564c0e6044e2 P 09608869c3374b659697b7b552d79c0a: No bootstrap required, opened a new log
I20260925 22:43:37.259723 16573 tablet_bootstrap.cc:492] T b999e5930df74d2787b7564c0e6044e2 P e0bc0bded1b14a7589d138d4bee9474b: No bootstrap required, opened a new log
I20260925 22:43:37.259934 16573 ts_tablet_manager.cc:1397] T b999e5930df74d2787b7564c0e6044e2 P e0bc0bded1b14a7589d138d4bee9474b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:43:37.259933 16571 ts_tablet_manager.cc:1397] T b999e5930df74d2787b7564c0e6044e2 P 09608869c3374b659697b7b552d79c0a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:43:37.261317 16572 tablet_bootstrap.cc:492] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b: No bootstrap required, opened a new log
I20260925 22:43:37.261586 16572 ts_tablet_manager.cc:1397] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:43:37.263281 16573 raft_consensus.cc:348] T b999e5930df74d2787b7564c0e6044e2 P e0bc0bded1b14a7589d138d4bee9474b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } }
I20260925 22:43:37.263610 16573 raft_consensus.cc:374] T b999e5930df74d2787b7564c0e6044e2 P e0bc0bded1b14a7589d138d4bee9474b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:43:37.263669 16573 raft_consensus.cc:729] T b999e5930df74d2787b7564c0e6044e2 P e0bc0bded1b14a7589d138d4bee9474b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0bc0bded1b14a7589d138d4bee9474b, State: Initialized, Role: FOLLOWER
I20260925 22:43:37.264211 16573 consensus_queue.cc:260] T b999e5930df74d2787b7564c0e6044e2 P e0bc0bded1b14a7589d138d4bee9474b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } }
I20260925 22:43:37.264711 16571 raft_consensus.cc:348] T b999e5930df74d2787b7564c0e6044e2 P 09608869c3374b659697b7b552d79c0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } }
I20260925 22:43:37.265008 16571 raft_consensus.cc:374] T b999e5930df74d2787b7564c0e6044e2 P 09608869c3374b659697b7b552d79c0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:43:37.265062 16571 raft_consensus.cc:729] T b999e5930df74d2787b7564c0e6044e2 P 09608869c3374b659697b7b552d79c0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09608869c3374b659697b7b552d79c0a, State: Initialized, Role: FOLLOWER
I20260925 22:43:37.265447 16573 ts_tablet_manager.cc:1428] T b999e5930df74d2787b7564c0e6044e2 P e0bc0bded1b14a7589d138d4bee9474b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 22:43:37.265532 16571 consensus_queue.cc:260] T b999e5930df74d2787b7564c0e6044e2 P 09608869c3374b659697b7b552d79c0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } }
I20260925 22:43:37.265738 16572 raft_consensus.cc:348] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } }
I20260925 22:43:37.266013 16420 heartbeater.cc:503] Master 127.2.74.254:34089 was elected leader, sending a full tablet report...
I20260925 22:43:37.266043 16572 raft_consensus.cc:374] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:43:37.266095 16572 raft_consensus.cc:729] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3193b1872a438f822322b5164b036b, State: Initialized, Role: FOLLOWER
I20260925 22:43:37.266314 16286 heartbeater.cc:503] Master 127.2.74.254:34089 was elected leader, sending a full tablet report...
I20260925 22:43:37.266572 16572 consensus_queue.cc:260] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } }
I20260925 22:43:37.266696 16571 ts_tablet_manager.cc:1428] T b999e5930df74d2787b7564c0e6044e2 P 09608869c3374b659697b7b552d79c0a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 22:43:37.267622 16572 ts_tablet_manager.cc:1428] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:43:37.267724 16554 heartbeater.cc:503] Master 127.2.74.254:34089 was elected leader, sending a full tablet report...
I20260925 22:43:37.274793 16579 raft_consensus.cc:482] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:43:37.275054 16579 raft_consensus.cc:504] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } }
I20260925 22:43:37.276253 16579 leader_election.cc:288] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0bc0bded1b14a7589d138d4bee9474b (127.2.74.194:39825), 09608869c3374b659697b7b552d79c0a (127.2.74.193:41583)
I20260925 22:43:37.280499 16556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:37.276521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:39825 (local address 127.2.74.195:46675)
0925 22:43:37.276653 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:37.276671 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:37.276779 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:37.277092 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:37.277094 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:37.277111 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:43:37.277255 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:37.277263 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.278153 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.278157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:37.279223 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:37.279234 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.279992 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.279996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:37.280118 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:37.280159 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 22:43:37.280237 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 22:43:37.280328 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":19}
I20260925 22:43:37.280524 16567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:37.276623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:46675 (local address 127.2.74.194:39825)
0925 22:43:37.276761 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:37.276766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:37.276784 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:43:37.276909 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:37.276912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:37.276974 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:37.277085 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:43:37.277377 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.278035 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.279355 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.279867 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.279973 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:37.280016 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:37.280266 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 22:43:37.280353 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:43:37.280711 16570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:37.276752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:34907 (local address 127.2.74.193:41583)
0925 22:43:37.276882 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:37.276887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:37.277215 (+ 328us) server_negotiation.cc:407] Connection header received
0925 22:43:37.277311 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:37.277316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:37.277376 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:37.277468 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:43:37.277753 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.278341 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.279599 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.280088 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.280224 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:37.280276 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:37.280478 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 22:43:37.280592 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:43:37.281245 16374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b999e5930df74d2787b7564c0e6044e2" candidate_uuid: "ac3193b1872a438f822322b5164b036b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0bc0bded1b14a7589d138d4bee9474b" is_pre_election: true
I20260925 22:43:37.281585 16374 raft_consensus.cc:2454] T b999e5930df74d2787b7564c0e6044e2 P e0bc0bded1b14a7589d138d4bee9474b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac3193b1872a438f822322b5164b036b in term 0.
I20260925 22:43:37.281876 16580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:37.276521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:41583 (local address 127.2.74.195:34907)
0925 22:43:37.277104 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:37.277120 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:37.277220 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:37.277503 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:37.277506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:37.277524 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:43:37.277637 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:37.277643 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.278481 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.278484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:37.279457 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:37.279470 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.280230 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.280236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:37.280344 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:37.280365 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:43:37.280445 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 22:43:37.280513 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":215,"thread_start_us":136,"threads_started":1}
I20260925 22:43:37.282194 16443 leader_election.cc:302] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac3193b1872a438f822322b5164b036b, e0bc0bded1b14a7589d138d4bee9474b; no voters:
I20260925 22:43:37.282541 16579 raft_consensus.cc:2781] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:43:37.282555 16240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b999e5930df74d2787b7564c0e6044e2" candidate_uuid: "ac3193b1872a438f822322b5164b036b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09608869c3374b659697b7b552d79c0a" is_pre_election: true
I20260925 22:43:37.282639 16579 raft_consensus.cc:482] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:43:37.282673 16579 raft_consensus.cc:3037] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:43:37.282859 16240 raft_consensus.cc:2454] T b999e5930df74d2787b7564c0e6044e2 P 09608869c3374b659697b7b552d79c0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac3193b1872a438f822322b5164b036b in term 0.
I20260925 22:43:37.284197 16579 raft_consensus.cc:504] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } }
I20260925 22:43:37.284636 16579 leader_election.cc:288] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b [CANDIDATE]: Term 1 election: Requested vote from peers e0bc0bded1b14a7589d138d4bee9474b (127.2.74.194:39825), 09608869c3374b659697b7b552d79c0a (127.2.74.193:41583)
I20260925 22:43:37.285127 16240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b999e5930df74d2787b7564c0e6044e2" candidate_uuid: "ac3193b1872a438f822322b5164b036b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09608869c3374b659697b7b552d79c0a"
I20260925 22:43:37.285127 16374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b999e5930df74d2787b7564c0e6044e2" candidate_uuid: "ac3193b1872a438f822322b5164b036b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0bc0bded1b14a7589d138d4bee9474b"
I20260925 22:43:37.285274 16240 raft_consensus.cc:3037] T b999e5930df74d2787b7564c0e6044e2 P 09608869c3374b659697b7b552d79c0a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:43:37.285274 16374 raft_consensus.cc:3037] T b999e5930df74d2787b7564c0e6044e2 P e0bc0bded1b14a7589d138d4bee9474b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:43:37.286674 16240 raft_consensus.cc:2454] T b999e5930df74d2787b7564c0e6044e2 P 09608869c3374b659697b7b552d79c0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac3193b1872a438f822322b5164b036b in term 1.
I20260925 22:43:37.286674 16374 raft_consensus.cc:2454] T b999e5930df74d2787b7564c0e6044e2 P e0bc0bded1b14a7589d138d4bee9474b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac3193b1872a438f822322b5164b036b in term 1.
I20260925 22:43:37.287068 16443 leader_election.cc:302] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac3193b1872a438f822322b5164b036b, e0bc0bded1b14a7589d138d4bee9474b; no voters:
I20260925 22:43:37.287287 16579 raft_consensus.cc:2781] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:43:37.287554 16579 raft_consensus.cc:686] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b [term 1 LEADER]: Becoming Leader. State: Replica: ac3193b1872a438f822322b5164b036b, State: Running, Role: LEADER
I20260925 22:43:37.287948 16579 consensus_queue.cc:237] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } }
I20260925 22:43:37.291455 16093 catalog_manager.cc:5495] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b reported cstate change: term changed from 0 to 1, leader changed from <none> to ac3193b1872a438f822322b5164b036b (127.2.74.195). New cstate: current_term: 1 leader_uuid: "ac3193b1872a438f822322b5164b036b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } health_report { overall_health: HEALTHY } } }
I20260925 22:43:37.305696 16569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:37.301940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46440 (local address 127.2.74.195:35695)
0925 22:43:37.302062 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:37.302065 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:37.302078 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:43:37.302192 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:37.302195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:37.302249 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:37.302338 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:43:37.302556 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.303141 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.304174 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.304385 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.304468 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:37.304787 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:43:37.305222 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:43:37.305316 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:37.305493 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 22:43:37.305560 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:43:37.311950 16240 raft_consensus.cc:1261] T b999e5930df74d2787b7564c0e6044e2 P 09608869c3374b659697b7b552d79c0a [term 1 FOLLOWER]: Refusing update from remote peer ac3193b1872a438f822322b5164b036b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:43:37.311951 16374 raft_consensus.cc:1261] T b999e5930df74d2787b7564c0e6044e2 P e0bc0bded1b14a7589d138d4bee9474b [term 1 FOLLOWER]: Refusing update from remote peer ac3193b1872a438f822322b5164b036b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:43:37.312775 16581 consensus_queue.cc:1035] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b [LEADER]: Connected to new peer: Peer: permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:43:37.313050 16579 consensus_queue.cc:1035] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:43:37.320565 16584 mvcc.cc:205] Tried to move back new op lower bound from 7333380986101997568 to 7333380986014539776. Current Snapshot: MvccSnapshot[applied={T|T < 7333380986101997568}]
I20260925 22:43:37.322856 16585 mvcc.cc:205] Tried to move back new op lower bound from 7333380986101997568 to 7333380986014539776. Current Snapshot: MvccSnapshot[applied={T|T < 7333380986101997568}]
I20260925 22:43:37.331846 16094 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:43:37.332072 16094 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:43:37.334929 16094 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af: Sending DeleteTablet for 3 replicas of tablet b999e5930df74d2787b7564c0e6044e2
I20260925 22:43:37.335861 16354 tablet_service.cc:1514] Processing DeleteTablet for tablet b999e5930df74d2787b7564c0e6044e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:43:37 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:41532
I20260925 22:43:37.335863 16488 tablet_service.cc:1514] Processing DeleteTablet for tablet b999e5930df74d2787b7564c0e6044e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:43:37 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:43968
I20260925 22:43:37.335950 16220 tablet_service.cc:1514] Processing DeleteTablet for tablet b999e5930df74d2787b7564c0e6044e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:43:37 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:38120
I20260925 22:43:37.336450 16594 tablet_replica.cc:330] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b: stopping tablet replica
I20260925 22:43:37.336500 16595 tablet_replica.cc:330] T b999e5930df74d2787b7564c0e6044e2 P e0bc0bded1b14a7589d138d4bee9474b: stopping tablet replica
I20260925 22:43:37.336743 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 16157
I20260925 22:43:37.337796 16595 raft_consensus.cc:2229] T b999e5930df74d2787b7564c0e6044e2 P e0bc0bded1b14a7589d138d4bee9474b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:43:37.338171 16595 raft_consensus.cc:2258] T b999e5930df74d2787b7564c0e6044e2 P e0bc0bded1b14a7589d138d4bee9474b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:43:37.336805 16594 raft_consensus.cc:2229] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:43:37.339600 16594 raft_consensus.cc:2258] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:43:37.340757 16595 ts_tablet_manager.cc:1905] T b999e5930df74d2787b7564c0e6044e2 P e0bc0bded1b14a7589d138d4bee9474b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:43:37.343143 16595 ts_tablet_manager.cc:1918] T b999e5930df74d2787b7564c0e6044e2 P e0bc0bded1b14a7589d138d4bee9474b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:43:37.343266 16595 log.cc:1196] T b999e5930df74d2787b7564c0e6044e2 P e0bc0bded1b14a7589d138d4bee9474b: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/wal/wals/b999e5930df74d2787b7564c0e6044e2
I20260925 22:43:37.343627 16595 ts_tablet_manager.cc:1939] T b999e5930df74d2787b7564c0e6044e2 P e0bc0bded1b14a7589d138d4bee9474b: Deleting consensus metadata
W20260925 22:43:37.344833 16082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.74.193:41583 (error 108)
I20260925 22:43:37.344988 16081 catalog_manager.cc:4841] TS e0bc0bded1b14a7589d138d4bee9474b (127.2.74.194:39825): tablet b999e5930df74d2787b7564c0e6044e2 (table test-table [id=d5800fa89f554c02abac5b96af45d1c2]) successfully deleted
I20260925 22:43:37.345062 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 16291
I20260925 22:43:37.347276 16594 ts_tablet_manager.cc:1905] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:43:37.349867 16594 ts_tablet_manager.cc:1918] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:43:37.350001 16594 log.cc:1196] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/wal/wals/b999e5930df74d2787b7564c0e6044e2
I20260925 22:43:37.350350 16594 ts_tablet_manager.cc:1939] T b999e5930df74d2787b7564c0e6044e2 P ac3193b1872a438f822322b5164b036b: Deleting consensus metadata
I20260925 22:43:37.351815 16080 catalog_manager.cc:4841] TS ac3193b1872a438f822322b5164b036b (127.2.74.195:35695): tablet b999e5930df74d2787b7564c0e6044e2 (table test-table [id=d5800fa89f554c02abac5b96af45d1c2]) successfully deleted
I20260925 22:43:37.352484 16566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:37.352175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:41583 (local address 127.0.0.1:52124)
0925 22:43:37.352315 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:37.352400 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.193:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 22:43:37.352707 16082 catalog_manager.cc:4584] TS 09608869c3374b659697b7b552d79c0a (127.2.74.193:41583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b999e5930df74d2787b7564c0e6044e2: Network error: Client connection negotiation failed: client connection to 127.2.74.193:41583: connect: Connection refused (error 111)
I20260925 22:43:37.353849 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 16425
I20260925 22:43:37.360230 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 16063
I20260925 22:43:37.367214 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:34089
--webserver_interface=127.2.74.254
--webserver_port=44589
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:34089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:43:37.483165 16598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:43:37.483533 16598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:43:37.483615 16598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:43:37.487888 16598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:43:37.488019 16598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:43:37.488066 16598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:43:37.488106 16598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:43:37.488142 16598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:43:37.493942 16598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:34089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:34089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=44589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:43:37.495245 16598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:43:37.496379 16598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:43:37.510313 16604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:37.510329 16607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:37.510370 16605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:37.510609 16598 server_base.cc:1034] running on GCE node
I20260925 22:43:37.511482 16598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:43:37.512782 16598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:43:37.513996 16598 hybrid_clock.cc:648] HybridClock initialized: now 1790376217513949 us; error 73 us; skew 500 ppm
Sep 25 22:43:37 dist-test-slave-csnl krb5kdc[16050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376217, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:43:37.517087 16598 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:43:37.518167 16598 webserver.cc:460] Webserver started at http://127.2.74.254:44589/ using document root <none> and password file <none>
I20260925 22:43:37.518785 16598 fs_manager.cc:359] Metadata directory not provided
I20260925 22:43:37.518865 16598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:43:37.523357 16598 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:43:37.525918 16614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:37.527431 16598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:43:37.527603 16598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "48ab6537b643474689184dfc76cba2af"
format_stamp: "Formatted at 2026-09-25 22:43:36 on dist-test-slave-csnl"
server_key: "279b47f6bc8d968eb07c9149acb6336f"
server_key_iv: "8b8674ac339906089d84ddbc4e68fcb9"
server_key_version: "encryptionkey@0"
I20260925 22:43:37.528306 16598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:43:37.553836 16598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:43:37.557199 16598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:43:37.557441 16598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:43:37.565343 16598 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:34089
I20260925 22:43:37.565646 16666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:34089 every 8 connection(s)
I20260925 22:43:37.566669 16598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:43:37.571626 16667 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:43:37.573628 16667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af: Bootstrap starting.
I20260925 22:43:37.574512 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 16598
I20260925 22:43:37.575083 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:41583
--local_ip_for_outbound_sockets=127.2.74.193
--tserver_master_addrs=127.2.74.254:34089
--webserver_port=46723
--webserver_interface=127.2.74.193
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:43:37.592185 16667 log.cc:824] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af: Log is configured to *not* fsync() on all Append() calls
I20260925 22:43:37.601410 16667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:43:37.601855 16667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af: Bootstrap complete.
I20260925 22:43:37.606402 16667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ab6537b643474689184dfc76cba2af" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34089 } }
I20260925 22:43:37.606906 16667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48ab6537b643474689184dfc76cba2af, State: Initialized, Role: FOLLOWER
I20260925 22:43:37.607506 16667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ab6537b643474689184dfc76cba2af" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34089 } }
I20260925 22:43:37.607677 16667 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:43:37.607784 16667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:43:37.607932 16667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:43:37.610672 16667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ab6537b643474689184dfc76cba2af" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34089 } }
I20260925 22:43:37.611158 16667 leader_election.cc:302] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48ab6537b643474689184dfc76cba2af; no voters:
I20260925 22:43:37.611585 16667 leader_election.cc:288] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 22:43:37.611757 16671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [term 2 FOLLOWER]: Leader election won for term 2
I20260925 22:43:37.612022 16671 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [term 2 LEADER]: Becoming Leader. State: Replica: 48ab6537b643474689184dfc76cba2af, State: Running, Role: LEADER
I20260925 22:43:37.612372 16671 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ab6537b643474689184dfc76cba2af" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34089 } }
I20260925 22:43:37.613054 16667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:43:37.613467 16673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "48ab6537b643474689184dfc76cba2af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ab6537b643474689184dfc76cba2af" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34089 } } }
I20260925 22:43:37.613523 16672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48ab6537b643474689184dfc76cba2af. Latest consensus state: current_term: 2 leader_uuid: "48ab6537b643474689184dfc76cba2af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ab6537b643474689184dfc76cba2af" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34089 } } }
I20260925 22:43:37.613610 16673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [sys.catalog]: This master's current role is: LEADER
I20260925 22:43:37.613633 16672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [sys.catalog]: This master's current role is: LEADER
I20260925 22:43:37.613974 16675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:43:37.619400 16675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:43:37.620093 16675 catalog_manager.cc:1259] Loaded cluster ID: fb6bdf8d2fbe47b49e4f57a10047c990
I20260925 22:43:37.620157 16675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:43:37.621589 16675 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 22:43:37.626317 16675 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 22:43:37.632723 16675 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:43:37.633764 16675 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af: Loaded TSK: 0
I20260925 22:43:37.635210 16675 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:43:37.706633 16669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:43:37.706895 16669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:43:37.706943 16669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:43:37.711581 16669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:43:37.711651 16669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:43:37.711740 16669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:43:37.717653 16669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:41583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=46723
--tserver_master_addrs=127.2.74.254:34089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:43:37.718868 16669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:43:37.720049 16669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:43:37.731137 16694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:37.731246 16695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:37.731446 16669 server_base.cc:1034] running on GCE node
W20260925 22:43:37.731462 16697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:37.732125 16669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:43:37.733028 16669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:43:37.734226 16669 hybrid_clock.cc:648] HybridClock initialized: now 1790376217734194 us; error 62 us; skew 500 ppm
Sep 25 22:43:37 dist-test-slave-csnl krb5kdc[16050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376217, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:43:37.737260 16669 init.cc:377] Logged in from keytab as kudu/127.2.74.193@KRBTEST.COM (short username kudu)
I20260925 22:43:37.738339 16669 webserver.cc:460] Webserver started at http://127.2.74.193:46723/ using document root <none> and password file <none>
I20260925 22:43:37.739001 16669 fs_manager.cc:359] Metadata directory not provided
I20260925 22:43:37.739158 16669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:43:37.743630 16669 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:43:37.746495 16704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:37.748035 16669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:43:37.748181 16669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "09608869c3374b659697b7b552d79c0a"
format_stamp: "Formatted at 2026-09-25 22:43:36 on dist-test-slave-csnl"
server_key: "06158e1ff88d038207b57438cbc1fa8d"
server_key_iv: "23cd8e23caaacc4f78f250f7cf905c35"
server_key_version: "encryptionkey@0"
I20260925 22:43:37.748804 16669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:43:37.761193 16669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:43:37.764663 16669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:43:37.764884 16669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:43:37.765582 16669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:43:37.766809 16711 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 22:43:37.771024 16669 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 22:43:37.771100 16669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260925 22:43:37.771153 16669 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 22:43:37.772923 16669 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 22:43:37.772990 16669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 22:43:37.773254 16711 tablet_bootstrap.cc:492] T b999e5930df74d2787b7564c0e6044e2 P 09608869c3374b659697b7b552d79c0a: Bootstrap starting.
I20260925 22:43:37.784355 16669 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:41583
I20260925 22:43:37.784368 16818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:41583 every 8 connection(s)
I20260925 22:43:37.785715 16669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:43:37.792359 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 16669
I20260925 22:43:37.792876 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:39825
--local_ip_for_outbound_sockets=127.2.74.194
--tserver_master_addrs=127.2.74.254:34089
--webserver_port=43337
--webserver_interface=127.2.74.194
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:43:37.794788 16711 log.cc:824] T b999e5930df74d2787b7564c0e6044e2 P 09608869c3374b659697b7b552d79c0a: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:43:37 dist-test-slave-csnl krb5kdc[16050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376217, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:43:37.804090 16711 tablet_bootstrap.cc:492] T b999e5930df74d2787b7564c0e6044e2 P 09608869c3374b659697b7b552d79c0a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:43:37.804533 16711 tablet_bootstrap.cc:492] T b999e5930df74d2787b7564c0e6044e2 P 09608869c3374b659697b7b552d79c0a: Bootstrap complete.
I20260925 22:43:37.805999 16711 ts_tablet_manager.cc:1397] T b999e5930df74d2787b7564c0e6044e2 P 09608869c3374b659697b7b552d79c0a: Time spent bootstrapping tablet: real 0.033s user 0.018s sys 0.011s
I20260925 22:43:37.807054 16822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:37.788156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.193:37825)
0925 22:43:37.788795 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:37.788841 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:37.789693 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:37.790865 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:37.790873 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:37.791360 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:43:37.791847 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:37.791863 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.793846 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.793849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:37.795302 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:37.795326 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.796241 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.796248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:37.796508 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:37.797273 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:43:37.797302 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:43:37.799453 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:43:37.802665 (+ 3212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:37.802672 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:37.802686 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:37.803131 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:37.803680 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:37.803683 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:37.803685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:37.803874 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:37.804773 (+ 899us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:43:37.804780 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:43:37.805205 (+ 425us) client_negotiation.cc:769] Sending connection context
0925 22:43:37.805414 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 22:43:37.805876 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":187,"threads_started":1}
I20260925 22:43:37.807431 16821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:37.788160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:37825 (local address 127.2.74.254:34089)
0925 22:43:37.788551 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:37.788574 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:37.788977 (+ 403us) server_negotiation.cc:407] Connection header received
0925 22:43:37.790140 (+ 1163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:37.790151 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:37.790466 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:37.790806 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:37.792082 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.793668 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.795471 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.796056 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.796254 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:37.799761 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:37.799773 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:37.799797 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:37.799847 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:37.802477 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:37.803305 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:37.803314 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:37.803324 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:37.803551 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:37.804003 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:37.804010 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:37.804014 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:37.804617 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:37.804776 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:37.805609 (+ 833us) server_negotiation.cc:299] Negotiation successful
0925 22:43:37.806411 (+ 802us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":144,"threads_started":1}
I20260925 22:43:37.809152 16711 raft_consensus.cc:348] T b999e5930df74d2787b7564c0e6044e2 P 09608869c3374b659697b7b552d79c0a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } }
I20260925 22:43:37.809689 16711 raft_consensus.cc:729] T b999e5930df74d2787b7564c0e6044e2 P 09608869c3374b659697b7b552d79c0a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09608869c3374b659697b7b552d79c0a, State: Initialized, Role: FOLLOWER
I20260925 22:43:37.810238 16711 consensus_queue.cc:260] T b999e5930df74d2787b7564c0e6044e2 P 09608869c3374b659697b7b552d79c0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } }
I20260925 22:43:37.811106 16819 heartbeater.cc:347] Connected to a master server at 127.2.74.254:34089
I20260925 22:43:37.811350 16819 heartbeater.cc:464] Registering TS with master...
I20260925 22:43:37.811426 16711 ts_tablet_manager.cc:1428] T b999e5930df74d2787b7564c0e6044e2 P 09608869c3374b659697b7b552d79c0a: Time spent starting tablet: real 0.005s user 0.006s sys 0.002s
I20260925 22:43:37.811920 16819 heartbeater.cc:511] Master 127.2.74.254:34089 requested a full tablet report, sending...
I20260925 22:43:37.814937 16631 ts_manager.cc:194] Registered new tserver with Master: 09608869c3374b659697b7b552d79c0a (127.2.74.193:41583)
I20260925 22:43:37.820001 16631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.193@KRBTEST.COM'} at 127.2.74.193:37825
I20260925 22:43:37.823068 16819 heartbeater.cc:503] Master 127.2.74.254:34089 was elected leader, sending a full tablet report...
Sep 25 22:43:37 dist-test-slave-csnl krb5kdc[16050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376217, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for kudu/127.2.74.193@KRBTEST.COM
I20260925 22:43:37.830543 16826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:37.818994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:41583 (local address 127.0.0.1:52126)
0925 22:43:37.819429 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:37.819482 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:37.819631 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:37.820359 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:37.820366 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:37.820752 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:43:37.820863 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:37.820870 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.822265 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.822269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:37.823248 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:37.823264 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.824093 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.824097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:37.824240 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:37.824896 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:43:37.824924 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:43:37.826379 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:43:37.828509 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:37.828523 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:37.828529 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:37.828804 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:37.829153 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:37.829156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:37.829158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:37.829256 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:37.829777 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:43:37.829784 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:43:37.829935 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 22:43:37.830085 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 22:43:37.830339 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":127,"threads_started":1}
I20260925 22:43:37.830543 16827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:37.819071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52126 (local address 127.2.74.193:41583)
0925 22:43:37.819875 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:37.819885 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:37.819905 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:43:37.819969 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:37.819973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:37.820182 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:37.820320 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:37.821009 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.822110 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.823429 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:37.823963 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:37.824093 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:37.826546 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:37.826558 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:37.826573 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:37.826608 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:37.828338 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:37.828944 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:37.828951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:37.828952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:37.829046 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:37.829351 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:37.829355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:37.829357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:37.829644 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:37.829802 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:37.830175 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 22:43:37.830357 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":702,"thread_start_us":107,"threads_started":1}
I20260925 22:43:37.831960 16753 tablet_service.cc:1514] Processing DeleteTablet for tablet b999e5930df74d2787b7564c0e6044e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:43:37 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:52126
I20260925 22:43:37.833101 16828 tablet_replica.cc:330] stopping tablet replica
I20260925 22:43:37.833472 16828 raft_consensus.cc:2229] T b999e5930df74d2787b7564c0e6044e2 P 09608869c3374b659697b7b552d79c0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:43:37.833751 16828 raft_consensus.cc:2258] T b999e5930df74d2787b7564c0e6044e2 P 09608869c3374b659697b7b552d79c0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:43:37.834759 16828 ts_tablet_manager.cc:1905] T b999e5930df74d2787b7564c0e6044e2 P 09608869c3374b659697b7b552d79c0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:43:37.838699 16828 ts_tablet_manager.cc:1918] T b999e5930df74d2787b7564c0e6044e2 P 09608869c3374b659697b7b552d79c0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:43:37.838843 16828 log.cc:1196] T b999e5930df74d2787b7564c0e6044e2 P 09608869c3374b659697b7b552d79c0a: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/wal/wals/b999e5930df74d2787b7564c0e6044e2
I20260925 22:43:37.839159 16828 ts_tablet_manager.cc:1939] T b999e5930df74d2787b7564c0e6044e2 P 09608869c3374b659697b7b552d79c0a: Deleting consensus metadata
I20260925 22:43:37.841029 16618 catalog_manager.cc:4841] TS 09608869c3374b659697b7b552d79c0a (127.2.74.193:41583): tablet b999e5930df74d2787b7564c0e6044e2 (table test-table [id=d5800fa89f554c02abac5b96af45d1c2]) successfully deleted
W20260925 22:43:37.922307 16823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:43:37.922595 16823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:43:37.922640 16823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:43:37.926855 16823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:43:37.926934 16823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:43:37.927073 16823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:43:37.932701 16823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:39825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=43337
--tserver_master_addrs=127.2.74.254:34089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:43:37.933897 16823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:43:37.934999 16823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:43:37.944888 16833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:37.944839 16834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:37.944849 16836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:37.945211 16823 server_base.cc:1034] running on GCE node
I20260925 22:43:37.945845 16823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:43:37.946585 16823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:43:37.947781 16823 hybrid_clock.cc:648] HybridClock initialized: now 1790376217947729 us; error 78 us; skew 500 ppm
Sep 25 22:43:37 dist-test-slave-csnl krb5kdc[16050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376217, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:43:37.950738 16823 init.cc:377] Logged in from keytab as kudu/127.2.74.194@KRBTEST.COM (short username kudu)
I20260925 22:43:37.951822 16823 webserver.cc:460] Webserver started at http://127.2.74.194:43337/ using document root <none> and password file <none>
I20260925 22:43:37.952522 16823 fs_manager.cc:359] Metadata directory not provided
I20260925 22:43:37.952644 16823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:43:37.957077 16823 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260925 22:43:37.959476 16843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:37.960915 16823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:43:37.961087 16823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "e0bc0bded1b14a7589d138d4bee9474b"
format_stamp: "Formatted at 2026-09-25 22:43:36 on dist-test-slave-csnl"
server_key: "e72ef7651d2168a7f8b05e9c1cd879b4"
server_key_iv: "37282bdc1b17f279721d3597822be6fa"
server_key_version: "encryptionkey@0"
I20260925 22:43:37.961695 16823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:43:37.990284 16823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:43:37.993888 16823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:43:37.994117 16823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:43:37.994752 16823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:43:37.995744 16823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:43:37.995820 16823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:37.995905 16823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:43:37.995955 16823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:38.005594 16823 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:39825
I20260925 22:43:38.005610 16956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:39825 every 8 connection(s)
I20260925 22:43:38.006749 16823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:43:38.010809 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 16823
I20260925 22:43:38.011308 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:35695
--local_ip_for_outbound_sockets=127.2.74.195
--tserver_master_addrs=127.2.74.254:34089
--webserver_port=37089
--webserver_interface=127.2.74.195
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:43:38 dist-test-slave-csnl krb5kdc[16050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376217, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:43:38.024291 16821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:38.008781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:50253 (local address 127.2.74.254:34089)
0925 22:43:38.008960 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:38.008964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:38.010172 (+ 1208us) server_negotiation.cc:407] Connection header received
0925 22:43:38.011344 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:38.011349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:38.011423 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:38.011527 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:38.013205 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.013928 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.016252 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.016500 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.016602 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:38.020170 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:38.020174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:38.020176 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:38.020207 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:38.021798 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:38.022444 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:38.022448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:38.022449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:38.022531 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:38.022925 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:38.022929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:38.022930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:38.023226 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:38.023352 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:38.023932 (+ 580us) server_negotiation.cc:299] Negotiation successful
0925 22:43:38.024103 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 22:43:38.025184 16959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:38.009181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.194:50253)
0925 22:43:38.009995 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:38.010059 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:38.011095 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:38.011774 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:38.011782 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:38.012198 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:43:38.012997 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:38.013022 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.014090 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.014098 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:38.016070 (+ 1972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:38.016089 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.016718 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.016727 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:38.017026 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:38.017826 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:43:38.017857 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:43:38.019928 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:43:38.021957 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:38.021965 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:38.021987 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:38.022308 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:38.022631 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:38.022635 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:38.022637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:38.022813 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:38.023354 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:43:38.023363 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:43:38.023664 (+ 301us) client_negotiation.cc:769] Sending connection context
0925 22:43:38.023871 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 22:43:38.024249 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":123,"threads_started":1}
I20260925 22:43:38.026525 16957 heartbeater.cc:347] Connected to a master server at 127.2.74.254:34089
I20260925 22:43:38.026813 16957 heartbeater.cc:464] Registering TS with master...
I20260925 22:43:38.027410 16957 heartbeater.cc:511] Master 127.2.74.254:34089 requested a full tablet report, sending...
I20260925 22:43:38.028630 16631 ts_manager.cc:194] Registered new tserver with Master: e0bc0bded1b14a7589d138d4bee9474b (127.2.74.194:39825)
I20260925 22:43:38.029562 16631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.194@KRBTEST.COM'} at 127.2.74.194:50253
W20260925 22:43:38.137235 16960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:43:38.137545 16960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:43:38.137593 16960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:43:38.141873 16960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:43:38.141949 16960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:43:38.142014 16960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:43:38.147753 16960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:35695
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=37089
--tserver_master_addrs=127.2.74.254:34089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:43:38.148975 16960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:43:38.150079 16960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:43:38.160789 16965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:38.160809 16968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:38.160789 16966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:38.161075 16960 server_base.cc:1034] running on GCE node
I20260925 22:43:38.161620 16960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:43:38.162302 16960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:43:38.163476 16960 hybrid_clock.cc:648] HybridClock initialized: now 1790376218163443 us; error 45 us; skew 500 ppm
Sep 25 22:43:38 dist-test-slave-csnl krb5kdc[16050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376218, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:43:38.166518 16960 init.cc:377] Logged in from keytab as kudu/127.2.74.195@KRBTEST.COM (short username kudu)
I20260925 22:43:38.167572 16960 webserver.cc:460] Webserver started at http://127.2.74.195:37089/ using document root <none> and password file <none>
I20260925 22:43:38.168186 16960 fs_manager.cc:359] Metadata directory not provided
I20260925 22:43:38.168269 16960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:43:38.172669 16960 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:43:38.175153 16975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:38.176630 16960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:43:38.176808 16960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "ac3193b1872a438f822322b5164b036b"
format_stamp: "Formatted at 2026-09-25 22:43:37 on dist-test-slave-csnl"
server_key: "9dcfab03b37f82a8b3d27e9bf7f378c3"
server_key_iv: "636c8c5bab73c4b9d4581737f2e5abf8"
server_key_version: "encryptionkey@0"
I20260925 22:43:38.177449 16960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:43:38.197511 16960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:43:38.200923 16960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:43:38.201166 16960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:43:38.201797 16960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:43:38.202701 16960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:43:38.202778 16960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:38.202849 16960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:43:38.202900 16960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:38.212626 16960 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:35695
I20260925 22:43:38.212713 17088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:35695 every 8 connection(s)
I20260925 22:43:38.213891 16960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:43:38.219547 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 16960
Sep 25 22:43:38 dist-test-slave-csnl krb5kdc[16050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376218, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:43:38.229545 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:38.220348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57540 (local address 127.2.74.254:34089)
0925 22:43:38.220635 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:38.220640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:38.220797 (+ 157us) server_negotiation.cc:407] Connection header received
0925 22:43:38.220952 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:38.220954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:38.221000 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:38.221075 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:38.222025 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.222775 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.223869 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.224196 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.224307 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:38.225858 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:38.225861 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:38.225863 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:38.225891 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:38.227553 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:38.228085 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:38.228090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:38.228093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:38.228198 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:38.228550 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:38.228553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:38.228555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:38.228778 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:38.229072 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:38.229183 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 22:43:38.229316 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"spinlock_wait_cycles":7936,"thread_start_us":80,"threads_started":1}
I20260925 22:43:38.231432 16821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:38.216191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:38169 (local address 127.2.74.254:34089)
0925 22:43:38.216373 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:38.216377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:38.217452 (+ 1075us) server_negotiation.cc:407] Connection header received
0925 22:43:38.218804 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:38.218807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:38.218862 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:38.218993 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:38.220878 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.221710 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.223423 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.223758 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.223872 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:38.227096 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:38.227100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:38.227102 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:38.227140 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:38.228758 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:38.229532 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:38.229536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:38.229548 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:38.229673 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:38.230130 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:38.230133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:38.230135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:38.230347 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:38.230452 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:38.231125 (+ 673us) server_negotiation.cc:299] Negotiation successful
0925 22:43:38.231261 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 22:43:38.232367 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:38.216681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.195:38169)
0925 22:43:38.217267 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:38.217316 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:38.218566 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:38.219260 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:38.219271 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:38.219861 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:43:38.220652 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:38.220679 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.221872 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.221875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:38.223285 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:38.223296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.223934 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.223938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:38.224149 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:38.224922 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:43:38.224951 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:43:38.226820 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:43:38.228902 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:38.228916 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:38.228936 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:38.229370 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:38.229799 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:38.229802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:38.229804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:38.230015 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:38.230484 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:43:38.230491 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:43:38.230818 (+ 327us) client_negotiation.cc:769] Sending connection context
0925 22:43:38.231068 (+ 250us) client_negotiation.cc:240] Negotiation successful
0925 22:43:38.231372 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":123,"threads_started":1}
I20260925 22:43:38.233572 17089 heartbeater.cc:347] Connected to a master server at 127.2.74.254:34089
I20260925 22:43:38.233860 17089 heartbeater.cc:464] Registering TS with master...
I20260925 22:43:38.234406 17089 heartbeater.cc:511] Master 127.2.74.254:34089 requested a full tablet report, sending...
I20260925 22:43:38.235612 16631 ts_manager.cc:194] Registered new tserver with Master: ac3193b1872a438f822322b5164b036b (127.2.74.195:35695)
I20260925 22:43:38.236205 16631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.195@KRBTEST.COM'} at 127.2.74.195:38169
I20260925 22:43:38.244041 2347 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:43:38.248633 16821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:38.244752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57548 (local address 127.2.74.254:34089)
0925 22:43:38.244898 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:38.244902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:38.245137 (+ 235us) server_negotiation.cc:407] Connection header received
0925 22:43:38.245275 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:38.245277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:38.245415 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:38.245506 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:43:38.245719 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.246225 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.247184 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.247417 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.247506 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:38.247762 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:43:38.248205 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:43:38.248299 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:38.248353 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 22:43:38.248431 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:43:38.249322 16631 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:43:38.252921 16631 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:43:38.261217 16753 tablet_service.cc:1467] Processing CreateTablet for tablet 19d246b090994cc183c30dbb1d7577bd (DEFAULT_TABLE table=test-table [id=ab70d532c83f4d06babaf8bbcda932d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:43:38.261853 16753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d246b090994cc183c30dbb1d7577bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:43:38.264828 16711 tablet_bootstrap.cc:492] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a: Bootstrap starting.
I20260925 22:43:38.265791 16711 tablet_bootstrap.cc:654] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a: Neither blocks nor log segments found. Creating new log.
I20260925 22:43:38.266559 16826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:38.260123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:39825 (local address 127.0.0.1:35498)
0925 22:43:38.260266 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:38.260283 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:38.260380 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:38.261416 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:38.261419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:38.261437 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:43:38.261574 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:38.261581 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.263406 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.263410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:38.264676 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:38.264687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.265948 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.265952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:38.266069 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:38.266132 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 22:43:38.266228 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 22:43:38.266367 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":18}
I20260925 22:43:38.266559 17098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:38.260324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35498 (local address 127.2.74.194:39825)
0925 22:43:38.260842 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:38.260855 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:38.260895 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:43:38.260985 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:38.260995 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:38.261216 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:38.261731 (+ 515us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:43:38.261923 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.263218 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.264863 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.265770 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.265979 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:38.266172 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:38.266271 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 22:43:38.266394 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":180,"threads_started":1}
I20260925 22:43:38.266810 17099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:38.260613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:35695 (local address 127.0.0.1:46444)
0925 22:43:38.260869 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:38.260883 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:38.260969 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:38.261944 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:38.261947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:38.261966 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:43:38.262081 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:38.262088 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.264100 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.264103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:38.265208 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:38.265219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.266408 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.266411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:38.266505 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:38.266529 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:43:38.266583 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 22:43:38.266665 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":4,"thread_start_us":114,"threads_started":1}
I20260925 22:43:38.267443 17100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:38.261054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46444 (local address 127.2.74.195:35695)
0925 22:43:38.261457 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:38.261471 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:38.261492 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:43:38.261563 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:38.261568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:38.261769 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:38.261926 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:43:38.262216 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.263535 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.265366 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.266259 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.266988 (+ 729us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:38.267144 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:38.267200 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 22:43:38.267292 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":164,"threads_started":1}
I20260925 22:43:38.267722 16711 tablet_bootstrap.cc:492] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a: No bootstrap required, opened a new log
I20260925 22:43:38.267819 16711 ts_tablet_manager.cc:1397] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:43:38.268292 16711 raft_consensus.cc:348] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } }
I20260925 22:43:38.268410 16711 raft_consensus.cc:374] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:43:38.268446 16711 raft_consensus.cc:729] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09608869c3374b659697b7b552d79c0a, State: Initialized, Role: FOLLOWER
I20260925 22:43:38.268582 16711 consensus_queue.cc:260] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } }
I20260925 22:43:38.268839 16711 ts_tablet_manager.cc:1428] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:43:38.270359 16891 tablet_service.cc:1467] Processing CreateTablet for tablet 19d246b090994cc183c30dbb1d7577bd (DEFAULT_TABLE table=test-table [id=ab70d532c83f4d06babaf8bbcda932d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:43:38.271620 16891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d246b090994cc183c30dbb1d7577bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:43:38.272323 17023 tablet_service.cc:1467] Processing CreateTablet for tablet 19d246b090994cc183c30dbb1d7577bd (DEFAULT_TABLE table=test-table [id=ab70d532c83f4d06babaf8bbcda932d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:43:38.273474 17023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d246b090994cc183c30dbb1d7577bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:43:38.279271 17102 tablet_bootstrap.cc:492] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b: Bootstrap starting.
I20260925 22:43:38.280620 17103 tablet_bootstrap.cc:492] T 19d246b090994cc183c30dbb1d7577bd P ac3193b1872a438f822322b5164b036b: Bootstrap starting.
I20260925 22:43:38.282034 17102 tablet_bootstrap.cc:654] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b: Neither blocks nor log segments found. Creating new log.
I20260925 22:43:38.283023 17102 log.cc:824] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:43:38.283449 17103 tablet_bootstrap.cc:654] T 19d246b090994cc183c30dbb1d7577bd P ac3193b1872a438f822322b5164b036b: Neither blocks nor log segments found. Creating new log.
I20260925 22:43:38.284142 17103 log.cc:824] T 19d246b090994cc183c30dbb1d7577bd P ac3193b1872a438f822322b5164b036b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:43:38.285089 17102 tablet_bootstrap.cc:492] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b: No bootstrap required, opened a new log
I20260925 22:43:38.285300 17102 ts_tablet_manager.cc:1397] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:43:38.285943 17103 tablet_bootstrap.cc:492] T 19d246b090994cc183c30dbb1d7577bd P ac3193b1872a438f822322b5164b036b: No bootstrap required, opened a new log
I20260925 22:43:38.286145 17103 ts_tablet_manager.cc:1397] T 19d246b090994cc183c30dbb1d7577bd P ac3193b1872a438f822322b5164b036b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
W20260925 22:43:38.287099 16820 tablet.cc:2369] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:43:38.288712 17102 raft_consensus.cc:348] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } }
I20260925 22:43:38.289060 17102 raft_consensus.cc:374] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:43:38.289124 17102 raft_consensus.cc:729] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0bc0bded1b14a7589d138d4bee9474b, State: Initialized, Role: FOLLOWER
I20260925 22:43:38.289546 17102 consensus_queue.cc:260] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } }
I20260925 22:43:38.289499 17103 raft_consensus.cc:348] T 19d246b090994cc183c30dbb1d7577bd P ac3193b1872a438f822322b5164b036b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } }
I20260925 22:43:38.289775 17103 raft_consensus.cc:374] T 19d246b090994cc183c30dbb1d7577bd P ac3193b1872a438f822322b5164b036b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:43:38.289834 17103 raft_consensus.cc:729] T 19d246b090994cc183c30dbb1d7577bd P ac3193b1872a438f822322b5164b036b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3193b1872a438f822322b5164b036b, State: Initialized, Role: FOLLOWER
I20260925 22:43:38.290267 16957 heartbeater.cc:503] Master 127.2.74.254:34089 was elected leader, sending a full tablet report...
I20260925 22:43:38.290287 17103 consensus_queue.cc:260] T 19d246b090994cc183c30dbb1d7577bd P ac3193b1872a438f822322b5164b036b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } }
I20260925 22:43:38.290624 17102 ts_tablet_manager.cc:1428] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 22:43:38.291016 17089 heartbeater.cc:503] Master 127.2.74.254:34089 was elected leader, sending a full tablet report...
I20260925 22:43:38.291293 17103 ts_tablet_manager.cc:1428] T 19d246b090994cc183c30dbb1d7577bd P ac3193b1872a438f822322b5164b036b: Time spent starting tablet: real 0.005s user 0.004s sys 0.003s
I20260925 22:43:38.353950 17106 raft_consensus.cc:482] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:43:38.354194 17106 raft_consensus.cc:504] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } }
I20260925 22:43:38.355552 16825 raft_consensus.cc:482] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:43:38.355597 17106 leader_election.cc:288] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac3193b1872a438f822322b5164b036b (127.2.74.195:35695), 09608869c3374b659697b7b552d79c0a (127.2.74.193:41583)
I20260925 22:43:38.355785 16825 raft_consensus.cc:504] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } }
I20260925 22:43:38.357129 16825 leader_election.cc:288] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0bc0bded1b14a7589d138d4bee9474b (127.2.74.194:39825), ac3193b1872a438f822322b5164b036b (127.2.74.195:35695)
I20260925 22:43:38.360747 16959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:38.355834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:35695 (local address 127.2.74.194:53103)
0925 22:43:38.356150 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:38.356167 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:38.356273 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:38.356894 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:38.356897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:38.356914 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:43:38.357034 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:38.357054 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.358017 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.358020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:38.359315 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:38.359326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.360295 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.360298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:38.360406 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:38.360429 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:43:38.360499 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 22:43:38.360591 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":1}
I20260925 22:43:38.361220 17100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:38.356403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:53103 (local address 127.2.74.195:35695)
0925 22:43:38.356565 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:38.356570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:38.356586 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:43:38.356642 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:38.356645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:38.356694 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:38.356770 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:43:38.357185 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.357909 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.359443 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.360045 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.360158 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:38.360234 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:38.361032 (+ 798us) server_negotiation.cc:299] Negotiation successful
0925 22:43:38.361113 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 22:43:38.361704 17109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:38.356035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:48897 (local address 127.2.74.193:41583)
0925 22:43:38.356274 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:38.356281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:38.356590 (+ 309us) server_negotiation.cc:407] Connection header received
0925 22:43:38.356770 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:38.356776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:38.356839 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:38.356937 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:43:38.357385 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.358118 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.359812 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.360506 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.360612 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:38.360763 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:38.361461 (+ 698us) server_negotiation.cc:299] Negotiation successful
0925 22:43:38.361548 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":166,"thread_start_us":78,"threads_started":1}
I20260925 22:43:38.361828 17043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d246b090994cc183c30dbb1d7577bd" candidate_uuid: "e0bc0bded1b14a7589d138d4bee9474b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac3193b1872a438f822322b5164b036b" is_pre_election: true
I20260925 22:43:38.362159 17043 raft_consensus.cc:2454] T 19d246b090994cc183c30dbb1d7577bd P ac3193b1872a438f822322b5164b036b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0bc0bded1b14a7589d138d4bee9474b in term 0.
I20260925 22:43:38.362250 17098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:38.357844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:55661 (local address 127.2.74.194:39825)
0925 22:43:38.357957 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:38.357961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:38.357973 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:43:38.358011 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:38.358014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:38.358049 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:38.358120 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:43:38.358544 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.359307 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.360833 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.361449 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.361529 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:38.361567 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:38.362003 (+ 436us) server_negotiation.cc:299] Negotiation successful
0925 22:43:38.362112 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":55}
I20260925 22:43:38.362478 17108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:38.355842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:41583 (local address 127.2.74.194:48897)
0925 22:43:38.356477 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:38.356491 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:38.356632 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:38.357083 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:38.357085 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:38.357101 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:43:38.357215 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:38.357224 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.358234 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.358237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:38.359677 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:38.359686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.361050 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.361054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:38.361149 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:38.361172 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:43:38.362011 (+ 839us) client_negotiation.cc:240] Negotiation successful
0925 22:43:38.362113 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":122,"thread_start_us":139,"threads_started":1}
I20260925 22:43:38.362633 17111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:38.357580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:35695 (local address 127.2.74.193:41627)
0925 22:43:38.358438 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:38.358452 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:38.358529 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:38.359056 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:38.359060 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:38.359076 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:43:38.359157 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:38.359162 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.360238 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.360243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:38.361351 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:38.361364 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.362221 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.362226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:38.362310 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:38.362334 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:43:38.362385 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 22:43:38.362479 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":3,"thread_start_us":76,"threads_started":1}
I20260925 22:43:38.362715 16845 leader_election.cc:302] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac3193b1872a438f822322b5164b036b, e0bc0bded1b14a7589d138d4bee9474b; no voters:
I20260925 22:43:38.362845 17110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:38.357453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:39825 (local address 127.2.74.193:55661)
0925 22:43:38.357803 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:38.357816 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:38.357895 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:38.358225 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:38.358228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:38.358246 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:43:38.358337 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:38.358342 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.359431 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.359434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:38.360705 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:38.360713 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.361670 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.361673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:38.361773 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:38.361871 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 22:43:38.362447 (+ 576us) client_negotiation.cc:240] Negotiation successful
0925 22:43:38.362548 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":39,"thread_start_us":88,"threads_started":1}
I20260925 22:43:38.363214 16773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d246b090994cc183c30dbb1d7577bd" candidate_uuid: "e0bc0bded1b14a7589d138d4bee9474b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09608869c3374b659697b7b552d79c0a" is_pre_election: true
I20260925 22:43:38.363306 17106 raft_consensus.cc:2781] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:43:38.363394 17112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:38.358150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:41627 (local address 127.2.74.195:35695)
0925 22:43:38.358790 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:38.358797 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:38.358809 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:43:38.358846 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:38.358849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:38.358889 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:38.359252 (+ 363us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:43:38.359347 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.360126 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.361493 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.362079 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.363097 (+ 1018us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:38.363139 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:38.363174 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 22:43:38.363236 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":121,"threads_started":1}
I20260925 22:43:38.363420 17106 raft_consensus.cc:482] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:43:38.363476 17106 raft_consensus.cc:3037] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:43:38.363502 16773 raft_consensus.cc:2454] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0bc0bded1b14a7589d138d4bee9474b in term 0.
I20260925 22:43:38.363659 16911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d246b090994cc183c30dbb1d7577bd" candidate_uuid: "09608869c3374b659697b7b552d79c0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0bc0bded1b14a7589d138d4bee9474b" is_pre_election: true
I20260925 22:43:38.363777 17043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d246b090994cc183c30dbb1d7577bd" candidate_uuid: "09608869c3374b659697b7b552d79c0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac3193b1872a438f822322b5164b036b" is_pre_election: true
I20260925 22:43:38.363902 17043 raft_consensus.cc:2454] T 19d246b090994cc183c30dbb1d7577bd P ac3193b1872a438f822322b5164b036b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09608869c3374b659697b7b552d79c0a in term 0.
I20260925 22:43:38.364274 16706 leader_election.cc:302] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09608869c3374b659697b7b552d79c0a, ac3193b1872a438f822322b5164b036b; no voters:
I20260925 22:43:38.364573 16825 raft_consensus.cc:2781] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:43:38.364660 16825 raft_consensus.cc:482] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:43:38.364706 16825 raft_consensus.cc:3037] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:43:38.364984 17106 raft_consensus.cc:504] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } }
I20260925 22:43:38.365314 16911 raft_consensus.cc:2379] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 09608869c3374b659697b7b552d79c0a in current term 1: Already voted for candidate e0bc0bded1b14a7589d138d4bee9474b in this term.
I20260925 22:43:38.365427 17106 leader_election.cc:288] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b [CANDIDATE]: Term 1 election: Requested vote from peers ac3193b1872a438f822322b5164b036b (127.2.74.195:35695), 09608869c3374b659697b7b552d79c0a (127.2.74.193:41583)
I20260925 22:43:38.365713 17043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d246b090994cc183c30dbb1d7577bd" candidate_uuid: "e0bc0bded1b14a7589d138d4bee9474b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac3193b1872a438f822322b5164b036b"
I20260925 22:43:38.365836 17043 raft_consensus.cc:3037] T 19d246b090994cc183c30dbb1d7577bd P ac3193b1872a438f822322b5164b036b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:43:38.365931 16773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d246b090994cc183c30dbb1d7577bd" candidate_uuid: "e0bc0bded1b14a7589d138d4bee9474b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09608869c3374b659697b7b552d79c0a"
I20260925 22:43:38.366122 16825 raft_consensus.cc:504] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } }
I20260925 22:43:38.366544 16825 leader_election.cc:288] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [CANDIDATE]: Term 1 election: Requested vote from peers e0bc0bded1b14a7589d138d4bee9474b (127.2.74.194:39825), ac3193b1872a438f822322b5164b036b (127.2.74.195:35695)
I20260925 22:43:38.366864 16773 raft_consensus.cc:2379] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e0bc0bded1b14a7589d138d4bee9474b in current term 1: Already voted for candidate 09608869c3374b659697b7b552d79c0a in this term.
I20260925 22:43:38.366920 17042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d246b090994cc183c30dbb1d7577bd" candidate_uuid: "09608869c3374b659697b7b552d79c0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac3193b1872a438f822322b5164b036b"
I20260925 22:43:38.367024 16911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d246b090994cc183c30dbb1d7577bd" candidate_uuid: "09608869c3374b659697b7b552d79c0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0bc0bded1b14a7589d138d4bee9474b"
I20260925 22:43:38.367189 16911 raft_consensus.cc:2379] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 09608869c3374b659697b7b552d79c0a in current term 1: Already voted for candidate e0bc0bded1b14a7589d138d4bee9474b in this term.
I20260925 22:43:38.367319 17043 raft_consensus.cc:2454] T 19d246b090994cc183c30dbb1d7577bd P ac3193b1872a438f822322b5164b036b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0bc0bded1b14a7589d138d4bee9474b in term 1.
I20260925 22:43:38.367539 16707 leader_election.cc:302] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09608869c3374b659697b7b552d79c0a; no voters: ac3193b1872a438f822322b5164b036b, e0bc0bded1b14a7589d138d4bee9474b
I20260925 22:43:38.367622 16845 leader_election.cc:302] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac3193b1872a438f822322b5164b036b, e0bc0bded1b14a7589d138d4bee9474b; no voters: 09608869c3374b659697b7b552d79c0a
I20260925 22:43:38.367751 16825 raft_consensus.cc:2726] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 22:43:38.367915 17106 raft_consensus.cc:2781] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:43:38.368150 17106 raft_consensus.cc:686] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b [term 1 LEADER]: Becoming Leader. State: Replica: e0bc0bded1b14a7589d138d4bee9474b, State: Running, Role: LEADER
I20260925 22:43:38.368534 17106 consensus_queue.cc:237] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } }
I20260925 22:43:38.371644 16631 catalog_manager.cc:5495] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b reported cstate change: term changed from 0 to 1, leader changed from <none> to e0bc0bded1b14a7589d138d4bee9474b (127.2.74.194). New cstate: current_term: 1 leader_uuid: "e0bc0bded1b14a7589d138d4bee9474b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } health_report { overall_health: UNKNOWN } } }
I20260925 22:43:38.398897 17098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:38.395184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35502 (local address 127.2.74.194:39825)
0925 22:43:38.395342 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:38.395347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:38.395362 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:43:38.395550 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:38.395554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:38.395612 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:38.395727 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:43:38.395972 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.396478 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.397350 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:38.397578 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:38.397685 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:38.398031 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:43:38.398486 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:43:38.398638 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:38.398673 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 22:43:38.398730 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 22:43:38.405936 16773 raft_consensus.cc:1261] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 1 FOLLOWER]: Refusing update from remote peer e0bc0bded1b14a7589d138d4bee9474b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:43:38.405936 17043 raft_consensus.cc:1261] T 19d246b090994cc183c30dbb1d7577bd P ac3193b1872a438f822322b5164b036b [term 1 FOLLOWER]: Refusing update from remote peer e0bc0bded1b14a7589d138d4bee9474b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:43:38.406679 17106 consensus_queue.cc:1035] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b [LEADER]: Connected to new peer: Peer: permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:43:38.406934 17113 consensus_queue.cc:1035] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:43:38.424068 16631 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:43:38.424283 16631 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:43:38.427152 16631 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af: Sending DeleteTablet for 3 replicas of tablet 19d246b090994cc183c30dbb1d7577bd
I20260925 22:43:38.428074 17023 tablet_service.cc:1514] Processing DeleteTablet for tablet 19d246b090994cc183c30dbb1d7577bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:43:38 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:46444
I20260925 22:43:38.428082 16891 tablet_service.cc:1514] Processing DeleteTablet for tablet 19d246b090994cc183c30dbb1d7577bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:43:38 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:35498
I20260925 22:43:38.428197 16753 tablet_service.cc:1514] Processing DeleteTablet for tablet 19d246b090994cc183c30dbb1d7577bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:43:38 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:52126
I20260925 22:43:38.428664 17127 tablet_replica.cc:330] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b: stopping tablet replica
I20260925 22:43:38.428848 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 16669
I20260925 22:43:38.428867 17128 tablet_replica.cc:330] T 19d246b090994cc183c30dbb1d7577bd P ac3193b1872a438f822322b5164b036b: stopping tablet replica
I20260925 22:43:38.429611 17128 raft_consensus.cc:2229] T 19d246b090994cc183c30dbb1d7577bd P ac3193b1872a438f822322b5164b036b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:43:38.430114 17128 raft_consensus.cc:2258] T 19d246b090994cc183c30dbb1d7577bd P ac3193b1872a438f822322b5164b036b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:43:38.430363 17127 raft_consensus.cc:2229] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:43:38.430773 17127 raft_consensus.cc:2258] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:43:38.432359 17127 ts_tablet_manager.cc:1905] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:43:38.432359 17128 ts_tablet_manager.cc:1905] T 19d246b090994cc183c30dbb1d7577bd P ac3193b1872a438f822322b5164b036b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:43:38.435526 17127 ts_tablet_manager.cc:1918] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:43:38.435526 17128 ts_tablet_manager.cc:1918] T 19d246b090994cc183c30dbb1d7577bd P ac3193b1872a438f822322b5164b036b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:43:38.435670 17127 log.cc:1196] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/wal/wals/19d246b090994cc183c30dbb1d7577bd
I20260925 22:43:38.435678 17128 log.cc:1196] T 19d246b090994cc183c30dbb1d7577bd P ac3193b1872a438f822322b5164b036b: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/wal/wals/19d246b090994cc183c30dbb1d7577bd
I20260925 22:43:38.436124 17127 ts_tablet_manager.cc:1939] T 19d246b090994cc183c30dbb1d7577bd P e0bc0bded1b14a7589d138d4bee9474b: Deleting consensus metadata
I20260925 22:43:38.436409 17128 ts_tablet_manager.cc:1939] T 19d246b090994cc183c30dbb1d7577bd P ac3193b1872a438f822322b5164b036b: Deleting consensus metadata
I20260925 22:43:38.437640 16616 catalog_manager.cc:4841] TS ac3193b1872a438f822322b5164b036b (127.2.74.195:35695): tablet 19d246b090994cc183c30dbb1d7577bd (table test-table [id=ab70d532c83f4d06babaf8bbcda932d2]) successfully deleted
I20260925 22:43:38.437640 16617 catalog_manager.cc:4841] TS e0bc0bded1b14a7589d138d4bee9474b (127.2.74.194:39825): tablet 19d246b090994cc183c30dbb1d7577bd (table test-table [id=ab70d532c83f4d06babaf8bbcda932d2]) successfully deleted
W20260925 22:43:38.441774 16618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.74.193:41583 (error 108)
I20260925 22:43:38.442008 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 16823
I20260925 22:43:38.443691 17099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:38.443233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:41583 (local address 127.0.0.1:52136)
0925 22:43:38.443489 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:38.443573 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.193:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
W20260925 22:43:38.443903 16618 catalog_manager.cc:4584] TS 09608869c3374b659697b7b552d79c0a (127.2.74.193:41583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 19d246b090994cc183c30dbb1d7577bd: Network error: Client connection negotiation failed: client connection to 127.2.74.193:41583: connect: Connection refused (error 111)
I20260925 22:43:38.448727 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 16960
I20260925 22:43:38.454917 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 16598
I20260925 22:43:38.461647 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:34089
--webserver_interface=127.2.74.254
--webserver_port=44589
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:34089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:43:38.578649 17130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:43:38.579094 17130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:43:38.579208 17130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:43:38.583704 17130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:43:38.583827 17130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:43:38.583865 17130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:43:38.583905 17130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:43:38.583946 17130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:43:38.589938 17130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:34089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:34089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=44589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:43:38.591251 17130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:43:38.592422 17130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:43:38.602413 17138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:38.602404 17136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:38.602403 17135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:38.602511 17130 server_base.cc:1034] running on GCE node
I20260925 22:43:38.603479 17130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:43:38.604779 17130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:43:38.606043 17130 hybrid_clock.cc:648] HybridClock initialized: now 1790376218606004 us; error 61 us; skew 500 ppm
Sep 25 22:43:38 dist-test-slave-csnl krb5kdc[16050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376218, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:43:38.615949 17130 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:43:38.617067 17130 webserver.cc:460] Webserver started at http://127.2.74.254:44589/ using document root <none> and password file <none>
I20260925 22:43:38.617733 17130 fs_manager.cc:359] Metadata directory not provided
I20260925 22:43:38.617841 17130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:43:38.622291 17130 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:43:38.624976 17145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:38.626475 17130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:43:38.626665 17130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "48ab6537b643474689184dfc76cba2af"
format_stamp: "Formatted at 2026-09-25 22:43:36 on dist-test-slave-csnl"
server_key: "279b47f6bc8d968eb07c9149acb6336f"
server_key_iv: "8b8674ac339906089d84ddbc4e68fcb9"
server_key_version: "encryptionkey@0"
I20260925 22:43:38.627420 17130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:43:38.657294 17130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:43:38.660943 17130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:43:38.661203 17130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:43:38.669198 17130 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:34089
I20260925 22:43:38.669545 17197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:34089 every 8 connection(s)
I20260925 22:43:38.670490 17130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:43:38.675235 17198 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:43:38.677110 17198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af: Bootstrap starting.
I20260925 22:43:38.679769 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 17130
I20260925 22:43:38.680260 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:41583
--local_ip_for_outbound_sockets=127.2.74.193
--tserver_master_addrs=127.2.74.254:34089
--webserver_port=46723
--webserver_interface=127.2.74.193
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:43:38.701138 17198 log.cc:824] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af: Log is configured to *not* fsync() on all Append() calls
I20260925 22:43:38.716606 17198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:43:38.717098 17198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af: Bootstrap complete.
I20260925 22:43:38.722615 17198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ab6537b643474689184dfc76cba2af" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34089 } }
I20260925 22:43:38.723204 17198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48ab6537b643474689184dfc76cba2af, State: Initialized, Role: FOLLOWER
I20260925 22:43:38.723834 17198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ab6537b643474689184dfc76cba2af" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34089 } }
I20260925 22:43:38.723994 17198 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:43:38.724082 17198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:43:38.724192 17198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [term 2 FOLLOWER]: Advancing to term 3
I20260925 22:43:38.727059 17198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ab6537b643474689184dfc76cba2af" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34089 } }
I20260925 22:43:38.727473 17198 leader_election.cc:302] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48ab6537b643474689184dfc76cba2af; no voters:
I20260925 22:43:38.727866 17198 leader_election.cc:288] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 22:43:38.728040 17202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [term 3 FOLLOWER]: Leader election won for term 3
I20260925 22:43:38.728338 17202 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [term 3 LEADER]: Becoming Leader. State: Replica: 48ab6537b643474689184dfc76cba2af, State: Running, Role: LEADER
I20260925 22:43:38.728729 17202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ab6537b643474689184dfc76cba2af" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34089 } }
I20260925 22:43:38.729291 17198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:43:38.730253 17203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "48ab6537b643474689184dfc76cba2af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ab6537b643474689184dfc76cba2af" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34089 } } }
I20260925 22:43:38.730433 17203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [sys.catalog]: This master's current role is: LEADER
I20260925 22:43:38.730768 17204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48ab6537b643474689184dfc76cba2af. Latest consensus state: current_term: 3 leader_uuid: "48ab6537b643474689184dfc76cba2af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ab6537b643474689184dfc76cba2af" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34089 } } }
I20260925 22:43:38.730861 17204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48ab6537b643474689184dfc76cba2af [sys.catalog]: This master's current role is: LEADER
I20260925 22:43:38.730990 17211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:43:38.736720 17211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:43:38.737795 17211 catalog_manager.cc:1259] Loaded cluster ID: fb6bdf8d2fbe47b49e4f57a10047c990
I20260925 22:43:38.737890 17211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 22:43:38.739004 17211 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc2511ef9ad google::LogMessage::Fail() at ??:0
@ 0x7fc2511f288a google::LogMessage::SendToLog() at ??:0
@ 0x7fc2511ef4de google::LogMessage::Flush() at ??:0
@ 0x7fc2511f0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc25bb2efdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fc25bb2fd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc25bb2c579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fc25bb70640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc2534413d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc253441c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fc2534433f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc253432512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc254a766db start_thread at ??:0
@ 0x7fc24fced61f clone at ??:0
W20260925 22:43:38.855137 17200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:43:38.855566 17200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:43:38.855652 17200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:43:38.862337 17200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:43:38.862442 17200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:43:38.862536 17200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:43:38.870920 17200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:41583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=46723
--tserver_master_addrs=127.2.74.254:34089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:43:38.872381 17200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:43:38.873732 17200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:43:38.885874 17242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:38.886016 17239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:38.885890 17240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:38.886180 17200 server_base.cc:1034] running on GCE node
I20260925 22:43:38.886665 17200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:43:38.887444 17200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:43:38.888634 17200 hybrid_clock.cc:648] HybridClock initialized: now 1790376218888604 us; error 48 us; skew 500 ppm
Sep 25 22:43:38 dist-test-slave-csnl krb5kdc[16050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376218, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:43:38.891637 17200 init.cc:377] Logged in from keytab as kudu/127.2.74.193@KRBTEST.COM (short username kudu)
I20260925 22:43:38.892647 17200 webserver.cc:460] Webserver started at http://127.2.74.193:46723/ using document root <none> and password file <none>
I20260925 22:43:38.893266 17200 fs_manager.cc:359] Metadata directory not provided
I20260925 22:43:38.893332 17200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:43:38.897768 17200 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:43:38.900386 17249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:38.901795 17200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:43:38.901937 17200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "09608869c3374b659697b7b552d79c0a"
format_stamp: "Formatted at 2026-09-25 22:43:36 on dist-test-slave-csnl"
server_key: "06158e1ff88d038207b57438cbc1fa8d"
server_key_iv: "23cd8e23caaacc4f78f250f7cf905c35"
server_key_version: "encryptionkey@0"
I20260925 22:43:38.902522 17200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:43:38.917282 17200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:43:38.920974 17200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:43:38.921211 17200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:43:38.921876 17200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:43:38.923120 17256 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 22:43:38.928915 17200 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 22:43:38.928998 17200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260925 22:43:38.929088 17200 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 22:43:38.930922 17200 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 22:43:38.931087 17200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.002s
I20260925 22:43:38.931414 17256 tablet_bootstrap.cc:492] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a: Bootstrap starting.
I20260925 22:43:38.942364 17200 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:41583
I20260925 22:43:38.943127 17363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:41583 every 8 connection(s)
I20260925 22:43:38.943629 17200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:43:38.948982 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 17200
I20260925 22:43:38.949434 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:39825
--local_ip_for_outbound_sockets=127.2.74.194
--tserver_master_addrs=127.2.74.254:34089
--webserver_port=43337
--webserver_interface=127.2.74.194
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:43:38.953856 17256 log.cc:824] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:43:38.963984 17256 tablet_bootstrap.cc:492] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:43:38.964496 17256 tablet_bootstrap.cc:492] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a: Bootstrap complete.
I20260925 22:43:38.966015 17256 ts_tablet_manager.cc:1397] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a: Time spent bootstrapping tablet: real 0.035s user 0.027s sys 0.004s
I20260925 22:43:38.969607 17256 raft_consensus.cc:348] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } }
I20260925 22:43:38.970227 17256 raft_consensus.cc:729] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09608869c3374b659697b7b552d79c0a, State: Initialized, Role: FOLLOWER
I20260925 22:43:38.970829 17256 consensus_queue.cc:260] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } }
I20260925 22:43:38.972043 17256 ts_tablet_manager.cc:1428] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260925 22:43:39.009444 17366 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:43:38.946350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.193:33543)
0925 22:43:38.946974 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:38.947043 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:38.948340 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:39.008502 (+ 60162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":143,"threads_started":1}
W20260925 22:43:39.009986 17364 heartbeater.cc:644] Failed to heartbeat to 127.2.74.254:34089 (0 consecutive failures): Network error: Failed to ping master at 127.2.74.254:34089: Client connection negotiation failed: client connection to 127.2.74.254:34089: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 22:43:39.010857 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:39.010584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.193:32933)
0925 22:43:39.010722 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:39.010793 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 22:43:39.011727 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:39.011498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.193:48015)
0925 22:43:39.011617 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:39.011674 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 22:43:39.011968 17364 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:43:39.073935 17367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:43:39.074294 17367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:43:39.074397 17367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:43:39.078872 17367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:43:39.079015 17367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:43:39.079138 17367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:43:39.085060 17367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:39825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=43337
--tserver_master_addrs=127.2.74.254:34089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:43:39.086315 17367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:43:39.087436 17367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:43:39.098029 17374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:39.098069 17377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:39.098069 17375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:39.098309 17367 server_base.cc:1034] running on GCE node
I20260925 22:43:39.098799 17367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:43:39.099586 17367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:43:39.100775 17367 hybrid_clock.cc:648] HybridClock initialized: now 1790376219100757 us; error 35 us; skew 500 ppm
Sep 25 22:43:39 dist-test-slave-csnl krb5kdc[16050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376219, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:43:39.104000 17367 init.cc:377] Logged in from keytab as kudu/127.2.74.194@KRBTEST.COM (short username kudu)
I20260925 22:43:39.105020 17367 webserver.cc:460] Webserver started at http://127.2.74.194:43337/ using document root <none> and password file <none>
I20260925 22:43:39.105651 17367 fs_manager.cc:359] Metadata directory not provided
I20260925 22:43:39.105722 17367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:43:39.110246 17367 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:43:39.112490 17384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:39.113819 17367 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260925 22:43:39.113979 17367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "e0bc0bded1b14a7589d138d4bee9474b"
format_stamp: "Formatted at 2026-09-25 22:43:36 on dist-test-slave-csnl"
server_key: "e72ef7651d2168a7f8b05e9c1cd879b4"
server_key_iv: "37282bdc1b17f279721d3597822be6fa"
server_key_version: "encryptionkey@0"
I20260925 22:43:39.114581 17367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:43:39.146570 17367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:43:39.150348 17367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:43:39.150537 17367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:43:39.151126 17367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:43:39.151971 17367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:43:39.152021 17367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:39.152063 17367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:43:39.152079 17367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:39.162114 17367 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:39825
I20260925 22:43:39.162134 17497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:39825 every 8 connection(s)
I20260925 22:43:39.163313 17367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:43:39.167305 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 17367
I20260925 22:43:39.167825 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:35695
--local_ip_for_outbound_sockets=127.2.74.195
--tserver_master_addrs=127.2.74.254:34089
--webserver_port=37089
--webserver_interface=127.2.74.195
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:43:39.168018 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:39.165736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.194:43867)
0925 22:43:39.166699 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:39.166835 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":748,"thread_start_us":570,"threads_started":1}
W20260925 22:43:39.168576 17498 heartbeater.cc:644] Failed to heartbeat to 127.2.74.254:34089 (0 consecutive failures): Network error: Failed to ping master at 127.2.74.254:34089: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
I20260925 22:43:39.169363 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:39.169020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.194:54465)
0925 22:43:39.169219 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:39.169296 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 22:43:39.170037 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:39.169818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.194:41365)
0925 22:43:39.169930 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:39.169988 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 22:43:39.170220 17498 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:43:39.283754 17501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:43:39.284075 17501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:43:39.284124 17501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:43:39.288527 17501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:43:39.288615 17501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:43:39.288726 17501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:43:39.294752 17501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:35695
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=37089
--tserver_master_addrs=127.2.74.254:34089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:43:39.296000 17501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:43:39.297144 17501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:43:39.308827 17507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:39.308930 17506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:39.308827 17509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:39.309152 17501 server_base.cc:1034] running on GCE node
I20260925 22:43:39.309638 17501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:43:39.310477 17501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:43:39.311653 17501 hybrid_clock.cc:648] HybridClock initialized: now 1790376219311632 us; error 40 us; skew 500 ppm
Sep 25 22:43:39 dist-test-slave-csnl krb5kdc[16050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376219, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:43:39.314786 17501 init.cc:377] Logged in from keytab as kudu/127.2.74.195@KRBTEST.COM (short username kudu)
I20260925 22:43:39.315835 17501 webserver.cc:460] Webserver started at http://127.2.74.195:37089/ using document root <none> and password file <none>
I20260925 22:43:39.316690 17501 fs_manager.cc:359] Metadata directory not provided
I20260925 22:43:39.316795 17501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:43:39.321447 17501 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260925 22:43:39.323886 17516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:39.325342 17501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:43:39.325510 17501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "ac3193b1872a438f822322b5164b036b"
format_stamp: "Formatted at 2026-09-25 22:43:37 on dist-test-slave-csnl"
server_key: "9dcfab03b37f82a8b3d27e9bf7f378c3"
server_key_iv: "636c8c5bab73c4b9d4581737f2e5abf8"
server_key_version: "encryptionkey@0"
I20260925 22:43:39.326138 17501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:43:39.341954 17501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:43:39.345698 17501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:43:39.345940 17501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:43:39.346609 17501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:43:39.347570 17501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:43:39.347651 17501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:39.347730 17501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:43:39.347774 17501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:39.357273 17501 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:35695
I20260925 22:43:39.357290 17629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:35695 every 8 connection(s)
I20260925 22:43:39.358353 17501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:43:39.362453 17632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:39.360686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.195:49581)
0925 22:43:39.361170 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:39.361672 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":119,"threads_started":1}
W20260925 22:43:39.363107 17630 heartbeater.cc:644] Failed to heartbeat to 127.2.74.254:34089 (0 consecutive failures): Network error: Failed to ping master at 127.2.74.254:34089: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
I20260925 22:43:39.363755 17632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:39.363511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.195:34299)
0925 22:43:39.363634 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:39.363687 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 22:43:39.364375 17632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:39.364191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.195:42967)
0925 22:43:39.364285 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:39.364327 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 22:43:39.364562 17630 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 22:43:39.365720 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 17501
I20260925 22:43:40.013116 17634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:40.012634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.193:55045)
0925 22:43:40.012919 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:40.012996 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":75,"threads_started":1}
I20260925 22:43:40.171416 17635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:40.170869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.194:37873)
0925 22:43:40.171210 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:40.171285 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":138,"threads_started":1}
I20260925 22:43:40.365680 17636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:40.365177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.195:39905)
0925 22:43:40.365492 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:40.365567 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":107,"threads_started":1}
I20260925 22:43:40.745813 17637 raft_consensus.cc:482] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:43:40.746052 17637 raft_consensus.cc:504] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } }
I20260925 22:43:40.747330 17637 leader_election.cc:288] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e0bc0bded1b14a7589d138d4bee9474b (127.2.74.194:39825), ac3193b1872a438f822322b5164b036b (127.2.74.195:35695)
Sep 25 22:43:40 dist-test-slave-csnl krb5kdc[16050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376218, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for kudu/127.2.74.194@KRBTEST.COM
Sep 25 22:43:40 dist-test-slave-csnl krb5kdc[16050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376218, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for kudu/127.2.74.195@KRBTEST.COM
I20260925 22:43:40.762181 17638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:40.747539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:39825 (local address 127.2.74.193:52169)
0925 22:43:40.747850 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:40.747868 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:40.747989 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:40.749760 (+ 1771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:40.749767 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:40.750366 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:43:40.750883 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:40.750900 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:40.752295 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:40.752303 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:40.753241 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:40.753251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:40.754015 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:40.754018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:40.754245 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:40.755038 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:43:40.755056 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:43:40.756686 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:43:40.759038 (+ 2352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:40.759045 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:40.759063 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:40.759422 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:40.759825 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:40.759828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:40.759831 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:40.760044 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:40.760938 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:43:40.760946 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:43:40.761306 (+ 360us) client_negotiation.cc:769] Sending connection context
0925 22:43:40.761546 (+ 240us) client_negotiation.cc:240] Negotiation successful
0925 22:43:40.761965 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":191,"spinlock_wait_cycles":36352,"thread_start_us":69,"threads_started":1}
I20260925 22:43:40.762212 17641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:40.747891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:60943 (local address 127.2.74.195:35695)
0925 22:43:40.748258 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:40.748277 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:40.748308 (+ 31us) server_negotiation.cc:407] Connection header received
0925 22:43:40.748942 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:40.748947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:40.749360 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:40.749695 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:40.751135 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:40.752123 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:40.753378 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:40.753877 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:40.754062 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:40.757250 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:40.757254 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:40.757264 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:40.757295 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:40.759280 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:40.759802 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:40.759810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:40.759815 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:40.759981 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:40.760289 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:40.760292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:40.760294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:40.760742 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:40.760876 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:40.761706 (+ 830us) server_negotiation.cc:299] Negotiation successful
0925 22:43:40.762022 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":92,"threads_started":1}
I20260925 22:43:40.762212 17640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:40.747748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:52169 (local address 127.2.74.194:39825)
0925 22:43:40.748197 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:40.748223 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:40.748261 (+ 38us) server_negotiation.cc:407] Connection header received
0925 22:43:40.748941 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:40.748947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:40.749359 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:40.749691 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:40.751159 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:40.752126 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:40.753403 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:40.753877 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:40.754062 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:40.756988 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:40.756996 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:40.757015 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:40.757051 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:40.758868 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:40.759540 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:40.759549 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:40.759558 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:40.759715 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:40.760140 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:40.760150 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:40.760152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:40.760742 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:40.760876 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:40.761706 (+ 830us) server_negotiation.cc:299] Negotiation successful
0925 22:43:40.762021 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":97,"threads_started":1}
I20260925 22:43:40.762182 17639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:40.747637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:35695 (local address 127.2.74.193:60943)
0925 22:43:40.747978 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:40.747993 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:40.748116 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:40.750011 (+ 1895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:40.750014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:40.750352 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:43:40.750883 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:40.750900 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:40.752290 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:40.752293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:40.753239 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:40.753249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:40.754015 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:40.754018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:40.754266 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:40.755010 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:43:40.755036 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:43:40.757063 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:43:40.759394 (+ 2331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:40.759397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:40.759399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:40.759664 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:40.760091 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:40.760093 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:40.760095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:40.760185 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:40.760938 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:43:40.760946 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:43:40.761306 (+ 360us) client_negotiation.cc:769] Sending connection context
0925 22:43:40.761546 (+ 240us) client_negotiation.cc:240] Negotiation successful
0925 22:43:40.762018 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":287,"spinlock_wait_cycles":17920,"thread_start_us":67,"threads_started":1}
W20260925 22:43:40.765674 17252 leader_election.cc:341] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0bc0bded1b14a7589d138d4bee9474b (127.2.74.194:39825): Not found: Tablet not found: 19d246b090994cc183c30dbb1d7577bd
W20260925 22:43:40.765853 17251 leader_election.cc:341] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ac3193b1872a438f822322b5164b036b (127.2.74.195:35695): Not found: Tablet not found: 19d246b090994cc183c30dbb1d7577bd
I20260925 22:43:40.765946 17251 leader_election.cc:302] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09608869c3374b659697b7b552d79c0a; no voters: ac3193b1872a438f822322b5164b036b, e0bc0bded1b14a7589d138d4bee9474b
I20260925 22:43:40.766225 17637 raft_consensus.cc:2726] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:43:41.014300 17639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:41.014003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.193:43993)
0925 22:43:41.014148 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:41.014210 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 22:43:41.172933 17642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:41.172340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.194:60435)
0925 22:43:41.172684 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:41.172760 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":96,"threads_started":1}
I20260925 22:43:41.367002 17643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:41.366523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.195:44361)
0925 22:43:41.366802 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:41.366868 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":80,"threads_started":1}
I20260925 22:43:42.015751 17644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:42.015274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.193:33413)
0925 22:43:42.015581 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:42.015648 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":76,"threads_started":1}
I20260925 22:43:42.174292 17645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:42.173838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.194:39099)
0925 22:43:42.174124 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:42.174189 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":94,"threads_started":1}
I20260925 22:43:42.368299 17646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:42.367843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.195:54539)
0925 22:43:42.368115 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:42.368177 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":75,"threads_started":1}
I20260925 22:43:42.802498 17647 raft_consensus.cc:482] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:43:42.802804 17647 raft_consensus.cc:504] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } }
I20260925 22:43:42.803331 17647 leader_election.cc:288] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e0bc0bded1b14a7589d138d4bee9474b (127.2.74.194:39825), ac3193b1872a438f822322b5164b036b (127.2.74.195:35695)
W20260925 22:43:42.804268 17252 leader_election.cc:341] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0bc0bded1b14a7589d138d4bee9474b (127.2.74.194:39825): Not found: Tablet not found: 19d246b090994cc183c30dbb1d7577bd
W20260925 22:43:42.804376 17251 leader_election.cc:341] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ac3193b1872a438f822322b5164b036b (127.2.74.195:35695): Not found: Tablet not found: 19d246b090994cc183c30dbb1d7577bd
I20260925 22:43:42.804512 17251 leader_election.cc:302] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09608869c3374b659697b7b552d79c0a; no voters: ac3193b1872a438f822322b5164b036b, e0bc0bded1b14a7589d138d4bee9474b
I20260925 22:43:42.804704 17647 raft_consensus.cc:2726] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:43:43.017084 17648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:43.016628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.193:52033)
0925 22:43:43.016910 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:43.016981 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":88,"threads_started":1}
I20260925 22:43:43.175616 17649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:43.175161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.194:43863)
0925 22:43:43.175454 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:43.175516 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":80,"threads_started":1}
I20260925 22:43:43.369647 17650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:43.369161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.195:59523)
0925 22:43:43.369473 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:43.369544 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":75,"threads_started":1}
I20260925 22:43:44.018738 17651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:44.018145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.193:57707)
0925 22:43:44.018496 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:44.018561 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":125,"threads_started":1}
I20260925 22:43:44.177147 17652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:44.176670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.194:51475)
0925 22:43:44.176981 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:44.177046 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":86,"threads_started":1}
I20260925 22:43:44.371088 17653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:44.370538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.195:40653)
0925 22:43:44.370835 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:44.370897 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":80,"threads_started":1}
I20260925 22:43:44.397516 17654 raft_consensus.cc:482] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:43:44.397657 17654 raft_consensus.cc:504] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } }
I20260925 22:43:44.398212 17654 leader_election.cc:288] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e0bc0bded1b14a7589d138d4bee9474b (127.2.74.194:39825), ac3193b1872a438f822322b5164b036b (127.2.74.195:35695)
W20260925 22:43:44.399080 17252 leader_election.cc:341] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0bc0bded1b14a7589d138d4bee9474b (127.2.74.194:39825): Not found: Tablet not found: 19d246b090994cc183c30dbb1d7577bd
W20260925 22:43:44.399189 17251 leader_election.cc:341] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ac3193b1872a438f822322b5164b036b (127.2.74.195:35695): Not found: Tablet not found: 19d246b090994cc183c30dbb1d7577bd
I20260925 22:43:44.399299 17252 leader_election.cc:302] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09608869c3374b659697b7b552d79c0a; no voters: ac3193b1872a438f822322b5164b036b, e0bc0bded1b14a7589d138d4bee9474b
I20260925 22:43:44.399479 17654 raft_consensus.cc:2726] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:43:45.020205 17655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:45.019716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.193:56317)
0925 22:43:45.020031 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:45.020101 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":98,"threads_started":1}
I20260925 22:43:45.178658 17656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:45.178119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.194:57417)
0925 22:43:45.178481 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:45.178553 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":82,"threads_started":1}
I20260925 22:43:45.372457 17657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:45.371976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.195:60419)
0925 22:43:45.372288 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:45.372355 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":99,"threads_started":1}
I20260925 22:43:46.021744 17658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:46.021208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.193:42217)
0925 22:43:46.021573 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:46.021641 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":142,"threads_started":1}
I20260925 22:43:46.180182 17659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:46.179572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.194:48667)
0925 22:43:46.179999 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:46.180070 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":109,"threads_started":1}
I20260925 22:43:46.373973 17660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:46.373431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.195:47191)
0925 22:43:46.373803 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:46.373870 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":103,"threads_started":1}
I20260925 22:43:46.420692 17661 raft_consensus.cc:482] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:43:46.420841 17661 raft_consensus.cc:504] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } }
I20260925 22:43:46.421391 17661 leader_election.cc:288] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e0bc0bded1b14a7589d138d4bee9474b (127.2.74.194:39825), ac3193b1872a438f822322b5164b036b (127.2.74.195:35695)
W20260925 22:43:46.422195 17252 leader_election.cc:341] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0bc0bded1b14a7589d138d4bee9474b (127.2.74.194:39825): Not found: Tablet not found: 19d246b090994cc183c30dbb1d7577bd
W20260925 22:43:46.422322 17251 leader_election.cc:341] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ac3193b1872a438f822322b5164b036b (127.2.74.195:35695): Not found: Tablet not found: 19d246b090994cc183c30dbb1d7577bd
I20260925 22:43:46.422436 17251 leader_election.cc:302] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09608869c3374b659697b7b552d79c0a; no voters: ac3193b1872a438f822322b5164b036b, e0bc0bded1b14a7589d138d4bee9474b
I20260925 22:43:46.422600 17661 raft_consensus.cc:2726] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:43:47.023219 17662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:47.022624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.193:57121)
0925 22:43:47.023040 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:47.023116 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":106,"threads_started":1}
I20260925 22:43:47.181716 17663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:47.181166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.194:34639)
0925 22:43:47.181537 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:47.181614 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":94,"threads_started":1}
I20260925 22:43:47.375501 17664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:47.375015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.195:55371)
0925 22:43:47.375316 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:47.375385 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":91,"threads_started":1}
I20260925 22:43:48.024840 17665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:48.024263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.193:44877)
0925 22:43:48.024599 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:48.024687 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":96,"threads_started":1}
I20260925 22:43:48.183117 17666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:48.182567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.194:58255)
0925 22:43:48.182873 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:48.182941 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":89,"threads_started":1}
I20260925 22:43:48.376721 17667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:48.376284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.195:53367)
0925 22:43:48.376561 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:48.376622 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":80,"threads_started":1}
I20260925 22:43:49.026563 17668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:49.025918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.193:48033)
0925 22:43:49.026315 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:49.026422 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":147,"threads_started":1}
I20260925 22:43:49.184437 17669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:49.184017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.194:41577)
0925 22:43:49.184279 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:49.184340 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":79,"threads_started":1}
I20260925 22:43:49.378080 17670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:49.377633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.195:36957)
0925 22:43:49.377909 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:49.377976 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":86,"threads_started":1}
I20260925 22:43:50.028126 17671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:50.027632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.193:51319)
0925 22:43:50.027946 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:50.028017 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":93,"threads_started":1}
I20260925 22:43:50.185870 17672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:50.185379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.194:43595)
0925 22:43:50.185695 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:50.185766 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":84,"threads_started":1}
I20260925 22:43:50.379586 17673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:50.379084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.195:46675)
0925 22:43:50.379408 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:50.379475 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":81,"threads_started":1}
I20260925 22:43:51.029608 17674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:51.029102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.193:47617)
0925 22:43:51.029394 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:51.029473 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":99,"threads_started":1}
I20260925 22:43:51.187132 17675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:51.186677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.194:40687)
0925 22:43:51.186920 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:51.187031 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":80,"threads_started":1}
I20260925 22:43:51.380929 17676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:51.380470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.195:49597)
0925 22:43:51.380758 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:51.380828 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":78,"threads_started":1}
I20260925 22:43:51.600673 17677 raft_consensus.cc:482] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:43:51.600826 17677 raft_consensus.cc:504] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bc0bded1b14a7589d138d4bee9474b" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39825 } } peers { permanent_uuid: "ac3193b1872a438f822322b5164b036b" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 35695 } } peers { permanent_uuid: "09608869c3374b659697b7b552d79c0a" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 41583 } }
I20260925 22:43:51.601238 17677 leader_election.cc:288] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e0bc0bded1b14a7589d138d4bee9474b (127.2.74.194:39825), ac3193b1872a438f822322b5164b036b (127.2.74.195:35695)
W20260925 22:43:51.602073 17251 leader_election.cc:341] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ac3193b1872a438f822322b5164b036b (127.2.74.195:35695): Not found: Tablet not found: 19d246b090994cc183c30dbb1d7577bd
W20260925 22:43:51.602190 17252 leader_election.cc:341] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0bc0bded1b14a7589d138d4bee9474b (127.2.74.194:39825): Not found: Tablet not found: 19d246b090994cc183c30dbb1d7577bd
I20260925 22:43:51.602296 17252 leader_election.cc:302] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09608869c3374b659697b7b552d79c0a; no voters: ac3193b1872a438f822322b5164b036b, e0bc0bded1b14a7589d138d4bee9474b
I20260925 22:43:51.602470 17677 raft_consensus.cc:2726] T 19d246b090994cc183c30dbb1d7577bd P 09608869c3374b659697b7b552d79c0a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:43:52.031136 17678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:52.030523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.193:36645)
0925 22:43:52.030837 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:52.030915 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":95,"threads_started":1}
I20260925 22:43:52.188526 17679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:52.187973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.194:50819)
0925 22:43:52.188324 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:52.188395 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":94,"threads_started":1}
I20260925 22:43:52.382285 17680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:52.381835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.195:53867)
0925 22:43:52.382120 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:52.382185 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":79,"threads_started":1}
I20260925 22:43:53.032442 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:53.031958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.193:35483)
0925 22:43:53.032264 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:53.032333 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":83,"threads_started":1}
I20260925 22:43:53.189958 17682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:53.189446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.194:42021)
0925 22:43:53.189787 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:53.189855 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":97,"threads_started":1}
I20260925 22:43:53.383688 17683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:53.383215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.195:53189)
0925 22:43:53.383487 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:53.383567 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":83,"threads_started":1}
I20260925 22:43:54.034025 17684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:54.033549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.193:46641)
0925 22:43:54.033856 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:54.033923 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":86,"threads_started":1}
I20260925 22:43:54.191641 17685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:54.191158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34089 (local address 127.2.74.194:40587)
0925 22:43:54.191466 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:54.191534 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":101,"threads_started":1}
I20260925 22:43:54.371798 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 17200
I20260925 22:43:54.378194 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 17367
I20260925 22:43:54.384047 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 17501
2026-09-25T22:43:54Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19324 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:43:54 dist-test-slave-csnl krb5kdc[17688](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:43:54 dist-test-slave-csnl krb5kdc[17688](info): set up 2 sockets
Sep 25 22:43:54 dist-test-slave-csnl krb5kdc[17688](info): commencing operation
krb5kdc: starting...
W20260925 22:43:55.431322 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:43:55 dist-test-slave-csnl krb5kdc[17688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:43:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:43:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:43:55.558250 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:34049
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:43275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:34049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:43:55.675961 17701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:43:55.676319 17701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:43:55.676395 17701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:43:55.680629 17701 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 22:43:55.680742 17701 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 22:43:55.680773 17701 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 22:43:55.680804 17701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:43:55.680848 17701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:43:55.680889 17701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:43:55.680927 17701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:43:55.680963 17701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:43:55.686661 17701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:43275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:34049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:34049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:43:55.687942 17701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:43:55.689083 17701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:43:55.699040 17707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:55.699317 17701 server_base.cc:1034] running on GCE node
W20260925 22:43:55.699043 17709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:55.699049 17706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:55.700212 17701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:43:55.701561 17701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:43:55.702798 17701 hybrid_clock.cc:648] HybridClock initialized: now 1790376235702758 us; error 63 us; skew 500 ppm
Sep 25 22:43:55 dist-test-slave-csnl krb5kdc[17688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376235, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:43:55.705974 17701 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:43:55.707157 17701 webserver.cc:460] Webserver started at http://127.2.74.254:43585/ using document root <none> and password file <none>
I20260925 22:43:55.707813 17701 fs_manager.cc:359] Metadata directory not provided
I20260925 22:43:55.707918 17701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:43:55.708175 17701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:43:55.710490 17701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "0e78555668e843efbb9de814ef9dcbe9"
format_stamp: "Formatted at 2026-09-25 22:43:55 on dist-test-slave-csnl"
server_key: "e63f0d422b6e8cce8d9c5b6516d9fddd"
server_key_iv: "663e048941c2f3de57ed20ebb28df4a4"
server_key_version: "encryptionkey@0"
I20260925 22:43:55.711231 17701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "0e78555668e843efbb9de814ef9dcbe9"
format_stamp: "Formatted at 2026-09-25 22:43:55 on dist-test-slave-csnl"
server_key: "e63f0d422b6e8cce8d9c5b6516d9fddd"
server_key_iv: "663e048941c2f3de57ed20ebb28df4a4"
server_key_version: "encryptionkey@0"
I20260925 22:43:55.715639 17701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:43:55.718562 17716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:55.720223 17701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:43:55.720417 17701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "0e78555668e843efbb9de814ef9dcbe9"
format_stamp: "Formatted at 2026-09-25 22:43:55 on dist-test-slave-csnl"
server_key: "e63f0d422b6e8cce8d9c5b6516d9fddd"
server_key_iv: "663e048941c2f3de57ed20ebb28df4a4"
server_key_version: "encryptionkey@0"
I20260925 22:43:55.720563 17701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:43:55.736027 17701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:43:55.739699 17701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:43:55.739928 17701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:43:55.747992 17701 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:34049
I20260925 22:43:55.748327 17768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:34049 every 8 connection(s)
I20260925 22:43:55.749105 17701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:43:55.752224 17769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:43:55.755283 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 17701
I20260925 22:43:55.755399 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:43:55.755678 2347 external_mini_cluster.cc:1428] Setting key cc1527680144a6e4a7b6714f3cf3d7f7
I20260925 22:43:55.758438 17769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9: Bootstrap starting.
I20260925 22:43:55.761420 17769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9: Neither blocks nor log segments found. Creating new log.
I20260925 22:43:55.762485 17769 log.cc:824] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9: Log is configured to *not* fsync() on all Append() calls
I20260925 22:43:55.764995 17769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9: No bootstrap required, opened a new log
Sep 25 22:43:55 dist-test-slave-csnl krb5kdc[17688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:43:55.769784 17769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e78555668e843efbb9de814ef9dcbe9" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34049 } }
I20260925 22:43:55.770133 17769 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:43:55.770187 17769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e78555668e843efbb9de814ef9dcbe9, State: Initialized, Role: FOLLOWER
I20260925 22:43:55.770747 17769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e78555668e843efbb9de814ef9dcbe9" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34049 } }
I20260925 22:43:55.770932 17769 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:43:55.771080 17769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:43:55.771198 17769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:43:55.772524 17769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e78555668e843efbb9de814ef9dcbe9" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34049 } }
I20260925 22:43:55.772655 17772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:55.757171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43354 (local address 127.2.74.254:34049)
0925 22:43:55.757624 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:55.757640 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:55.757677 (+ 37us) server_negotiation.cc:407] Connection header received
0925 22:43:55.758340 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:55.758359 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:55.758773 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:55.759147 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:55.760120 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:55.761170 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:55.762447 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:55.763096 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:55.763237 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:55.766064 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:55.766072 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:55.766092 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:55.766135 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:55.769003 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:55.769561 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:55.769568 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:55.769575 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:55.769737 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:55.770022 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:55.770026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:55.770027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:55.770608 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:55.770827 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:55.771190 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 22:43:55.771550 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":160,"threads_started":1}
I20260925 22:43:55.772954 17769 leader_election.cc:302] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e78555668e843efbb9de814ef9dcbe9; no voters:
I20260925 22:43:55.773355 17769 leader_election.cc:288] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:43:55.773500 17774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:43:55.773757 17774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [term 1 LEADER]: Becoming Leader. State: Replica: 0e78555668e843efbb9de814ef9dcbe9, State: Running, Role: LEADER
I20260925 22:43:55.774202 17774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e78555668e843efbb9de814ef9dcbe9" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34049 } }
I20260925 22:43:55.774749 17769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:43:55.775532 17775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e78555668e843efbb9de814ef9dcbe9. Latest consensus state: current_term: 1 leader_uuid: "0e78555668e843efbb9de814ef9dcbe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e78555668e843efbb9de814ef9dcbe9" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34049 } } }
I20260925 22:43:55.775807 17775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [sys.catalog]: This master's current role is: LEADER
I20260925 22:43:55.776049 17776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e78555668e843efbb9de814ef9dcbe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e78555668e843efbb9de814ef9dcbe9" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34049 } } }
I20260925 22:43:55.776175 17776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [sys.catalog]: This master's current role is: LEADER
I20260925 22:43:55.776285 17783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:43:55.780331 17783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:43:55.787261 17783 catalog_manager.cc:1347] Generated new cluster ID: b95bcf3b1ddc4e09a3b9423c1f0db741
I20260925 22:43:55.787357 17783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:43:55.815644 17783 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:43:55.817068 17783 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:43:55.830327 17783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9: Generated new TSK 0
I20260925 22:43:55.831133 17783 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
I20260925 22:43:55.869709 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:0
--local_ip_for_outbound_sockets=127.2.74.193
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:34049
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:43275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:43:56.007305 17795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:43:56.007604 17795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:43:56.007691 17795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:43:56.012040 17795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:43:56.012122 17795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:43:56.012236 17795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:43:56.017800 17795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:43275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:34049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:43:56.019049 17795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:43:56.020198 17795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:43:56.030550 17800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:56.030563 17803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:56.030702 17795 server_base.cc:1034] running on GCE node
W20260925 22:43:56.030902 17801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:56.031369 17795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:43:56.032105 17795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:43:56.033313 17795 hybrid_clock.cc:648] HybridClock initialized: now 1790376236033290 us; error 35 us; skew 500 ppm
Sep 25 22:43:56 dist-test-slave-csnl krb5kdc[17688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376236, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:43:56.036497 17795 init.cc:377] Logged in from keytab as kudu/127.2.74.193@KRBTEST.COM (short username kudu)
I20260925 22:43:56.037539 17795 webserver.cc:460] Webserver started at http://127.2.74.193:44489/ using document root <none> and password file <none>
I20260925 22:43:56.038116 17795 fs_manager.cc:359] Metadata directory not provided
I20260925 22:43:56.038236 17795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:43:56.038548 17795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:43:56.041152 17795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/data/instance:
uuid: "4bed4a0e54674a99ba4661ddb1ce6395"
format_stamp: "Formatted at 2026-09-25 22:43:56 on dist-test-slave-csnl"
server_key: "62831131f6d0b5de48446fa4ad439ff8"
server_key_iv: "f5dbf9d44f655741b1882dbcd3840308"
server_key_version: "encryptionkey@0"
I20260925 22:43:56.041821 17795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance:
uuid: "4bed4a0e54674a99ba4661ddb1ce6395"
format_stamp: "Formatted at 2026-09-25 22:43:56 on dist-test-slave-csnl"
server_key: "62831131f6d0b5de48446fa4ad439ff8"
server_key_iv: "f5dbf9d44f655741b1882dbcd3840308"
server_key_version: "encryptionkey@0"
I20260925 22:43:56.046084 17795 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 22:43:56.049173 17810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:56.050617 17795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:43:56.050753 17795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "4bed4a0e54674a99ba4661ddb1ce6395"
format_stamp: "Formatted at 2026-09-25 22:43:56 on dist-test-slave-csnl"
server_key: "62831131f6d0b5de48446fa4ad439ff8"
server_key_iv: "f5dbf9d44f655741b1882dbcd3840308"
server_key_version: "encryptionkey@0"
I20260925 22:43:56.050913 17795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:43:56.065805 17795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:43:56.069408 17795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:43:56.069633 17795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:43:56.070283 17795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:43:56.071235 17795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:43:56.071286 17795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:56.071328 17795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:43:56.071378 17795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:56.081200 17795 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:37037
I20260925 22:43:56.081224 17923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:37037 every 8 connection(s)
I20260925 22:43:56.082151 17795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:43:56.087575 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 17795
I20260925 22:43:56.087730 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance
I20260925 22:43:56.088056 2347 external_mini_cluster.cc:1428] Setting key 48a93b1bdcfa9ff4626e458e8769b5d2
Sep 25 22:43:56 dist-test-slave-csnl krb5kdc[17688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376236, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.74.194@KRBTEST.COM; defaulting to no policy
I20260925 22:43:56.098171 17772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:56.084069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:40197 (local address 127.2.74.254:34049)
0925 22:43:56.084213 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:56.084216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:56.085084 (+ 868us) server_negotiation.cc:407] Connection header received
0925 22:43:56.086436 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:56.086440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:56.086505 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:56.086614 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:56.088308 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.088865 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.090363 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.090788 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.090921 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:56.094137 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:56.094141 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:56.094143 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:56.094171 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:56.095907 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:56.096506 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:56.096510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:56.096511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:56.096595 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:56.096951 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:56.096955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:56.096957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:56.097169 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:56.097278 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:56.097828 (+ 550us) server_negotiation.cc:299] Negotiation successful
0925 22:43:56.097980 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:43:56.099104 17926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:56.084424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.193:40197)
0925 22:43:56.084924 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:56.084970 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:56.086196 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:56.086832 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:56.086840 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:56.087319 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:43:56.088068 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:56.088093 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.089010 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.089014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:56.090198 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:56.090210 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.091016 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.091021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:56.091230 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:56.092008 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:43:56.092038 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:43:56.093909 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:43:56.096057 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:56.096070 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:56.096087 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:56.096378 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:56.096692 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:56.096695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:56.096697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:56.096853 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:56.097280 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:43:56.097290 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:43:56.097573 (+ 283us) client_negotiation.cc:769] Sending connection context
0925 22:43:56.097786 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 22:43:56.098157 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":131,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.194@KRBTEST.COM" created.
I20260925 22:43:56.100605 17924 heartbeater.cc:347] Connected to a master server at 127.2.74.254:34049
I20260925 22:43:56.100919 17924 heartbeater.cc:464] Registering TS with master...
I20260925 22:43:56.101464 17924 heartbeater.cc:511] Master 127.2.74.254:34049 requested a full tablet report, sending...
I20260925 22:43:56.103340 17733 ts_manager.cc:194] Registered new tserver with Master: 4bed4a0e54674a99ba4661ddb1ce6395 (127.2.74.193:37037)
I20260925 22:43:56.104671 17733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.193@KRBTEST.COM'} at 127.2.74.193:40197
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
I20260925 22:43:56.116402 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:0
--local_ip_for_outbound_sockets=127.2.74.194
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:34049
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:43275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:43:56.234141 17929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:43:56.234494 17929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:43:56.234573 17929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:43:56.238869 17929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:43:56.239035 17929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:43:56.239135 17929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:43:56.245078 17929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:43275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:34049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:43:56.246299 17929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:43:56.247546 17929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:43:56.258044 17934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:56.258033 17935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:56.258289 17929 server_base.cc:1034] running on GCE node
W20260925 22:43:56.258062 17937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:56.258909 17929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:43:56.259721 17929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:43:56.260893 17929 hybrid_clock.cc:648] HybridClock initialized: now 1790376236260856 us; error 57 us; skew 500 ppm
Sep 25 22:43:56 dist-test-slave-csnl krb5kdc[17688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376236, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:43:56.264062 17929 init.cc:377] Logged in from keytab as kudu/127.2.74.194@KRBTEST.COM (short username kudu)
I20260925 22:43:56.265059 17929 webserver.cc:460] Webserver started at http://127.2.74.194:33473/ using document root <none> and password file <none>
I20260925 22:43:56.265688 17929 fs_manager.cc:359] Metadata directory not provided
I20260925 22:43:56.265758 17929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:43:56.266016 17929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:43:56.268337 17929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/data/instance:
uuid: "c767841753fa44429e8c8185fa5114fd"
format_stamp: "Formatted at 2026-09-25 22:43:56 on dist-test-slave-csnl"
server_key: "bf3482e6a04f9d506d0dfede7a345283"
server_key_iv: "51fbcdda3594314272adb636f7511411"
server_key_version: "encryptionkey@0"
I20260925 22:43:56.268980 17929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance:
uuid: "c767841753fa44429e8c8185fa5114fd"
format_stamp: "Formatted at 2026-09-25 22:43:56 on dist-test-slave-csnl"
server_key: "bf3482e6a04f9d506d0dfede7a345283"
server_key_iv: "51fbcdda3594314272adb636f7511411"
server_key_version: "encryptionkey@0"
I20260925 22:43:56.273169 17929 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260925 22:43:56.276016 17944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:56.277467 17929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:43:56.277603 17929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "c767841753fa44429e8c8185fa5114fd"
format_stamp: "Formatted at 2026-09-25 22:43:56 on dist-test-slave-csnl"
server_key: "bf3482e6a04f9d506d0dfede7a345283"
server_key_iv: "51fbcdda3594314272adb636f7511411"
server_key_version: "encryptionkey@0"
I20260925 22:43:56.277729 17929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:43:56.296842 17929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:43:56.300487 17929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:43:56.300717 17929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:43:56.301379 17929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:43:56.302276 17929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:43:56.302325 17929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:56.302389 17929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:43:56.302423 17929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:56.312232 17929 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:39625
I20260925 22:43:56.312258 18057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:39625 every 8 connection(s)
I20260925 22:43:56.313257 17929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:43:56.323485 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 17929
Sep 25 22:43:56 dist-test-slave-csnl krb5kdc[17688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376236, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:43:56.323647 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance
I20260925 22:43:56.323954 2347 external_mini_cluster.cc:1428] Setting key 951ea8cc8a65b77a4727d4f4501e78a9
I20260925 22:43:56.329231 17772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:56.315294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:59649 (local address 127.2.74.254:34049)
0925 22:43:56.315442 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:56.315446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:56.316663 (+ 1217us) server_negotiation.cc:407] Connection header received
0925 22:43:56.317700 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:56.317703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:56.317771 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:56.317868 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:56.319215 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.319767 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.321041 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.321288 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.321401 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:56.324594 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:56.324597 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:56.324600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:56.324632 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:56.326486 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:56.327266 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:56.327273 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:56.327277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:56.327393 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:56.327927 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:56.327930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:56.327931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:56.328140 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:56.328230 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:56.328969 (+ 739us) server_negotiation.cc:299] Negotiation successful
0925 22:43:56.329096 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:43:56.330598 18060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:56.315738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.194:59649)
0925 22:43:56.316476 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:56.316524 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:56.317411 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:56.318068 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:56.318075 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:56.318474 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:43:56.319024 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:56.319040 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.319931 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.319935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:56.320887 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:56.320899 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.321471 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.321475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:56.321662 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:56.322317 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:43:56.322335 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:43:56.324325 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:43:56.326701 (+ 2376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:56.326708 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:56.326726 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:56.327148 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:56.327514 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:56.327519 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:56.327523 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:56.327782 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:56.328256 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:43:56.328264 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:43:56.328677 (+ 413us) client_negotiation.cc:769] Sending connection context
0925 22:43:56.328907 (+ 230us) client_negotiation.cc:240] Negotiation successful
0925 22:43:56.329311 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":121,"threads_started":1}
I20260925 22:43:56.332055 18058 heartbeater.cc:347] Connected to a master server at 127.2.74.254:34049
I20260925 22:43:56.332345 18058 heartbeater.cc:464] Registering TS with master...
I20260925 22:43:56.332981 18058 heartbeater.cc:511] Master 127.2.74.254:34049 requested a full tablet report, sending...
I20260925 22:43:56.334297 17733 ts_manager.cc:194] Registered new tserver with Master: c767841753fa44429e8c8185fa5114fd (127.2.74.194:39625)
WARNING: no policy specified for kudu/127.2.74.195@KRBTEST.COM; defaulting to no policy
I20260925 22:43:56.335035 17733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.194@KRBTEST.COM'} at 127.2.74.194:59649
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
I20260925 22:43:56.354624 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:0
--local_ip_for_outbound_sockets=127.2.74.195
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:34049
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:43275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:43:56.470526 18063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:43:56.470822 18063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:43:56.470867 18063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:43:56.475095 18063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:43:56.475171 18063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:43:56.475274 18063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:43:56.480827 18063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:43275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:34049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:43:56.481976 18063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:43:56.483122 18063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:43:56.493062 18068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:56.493098 18071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:43:56.493096 18069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:43:56.493515 18063 server_base.cc:1034] running on GCE node
I20260925 22:43:56.493992 18063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:43:56.494752 18063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:43:56.495949 18063 hybrid_clock.cc:648] HybridClock initialized: now 1790376236495925 us; error 74 us; skew 500 ppm
Sep 25 22:43:56 dist-test-slave-csnl krb5kdc[17688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376236, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:43:56.499063 18063 init.cc:377] Logged in from keytab as kudu/127.2.74.195@KRBTEST.COM (short username kudu)
I20260925 22:43:56.500059 18063 webserver.cc:460] Webserver started at http://127.2.74.195:36811/ using document root <none> and password file <none>
I20260925 22:43:56.500731 18063 fs_manager.cc:359] Metadata directory not provided
I20260925 22:43:56.500826 18063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:43:56.501066 18063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:43:56.503319 18063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/data/instance:
uuid: "c0a24c924c484ac2be5dbec2b5bb45e1"
format_stamp: "Formatted at 2026-09-25 22:43:56 on dist-test-slave-csnl"
server_key: "df8405a897aaa0f23d64580c402fbd5f"
server_key_iv: "ec828e616be0374ba2f487fba9699aab"
server_key_version: "encryptionkey@0"
I20260925 22:43:56.503933 18063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance:
uuid: "c0a24c924c484ac2be5dbec2b5bb45e1"
format_stamp: "Formatted at 2026-09-25 22:43:56 on dist-test-slave-csnl"
server_key: "df8405a897aaa0f23d64580c402fbd5f"
server_key_iv: "ec828e616be0374ba2f487fba9699aab"
server_key_version: "encryptionkey@0"
I20260925 22:43:56.507925 18063 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:43:56.510531 18078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:56.511953 18063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:43:56.512125 18063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "c0a24c924c484ac2be5dbec2b5bb45e1"
format_stamp: "Formatted at 2026-09-25 22:43:56 on dist-test-slave-csnl"
server_key: "df8405a897aaa0f23d64580c402fbd5f"
server_key_iv: "ec828e616be0374ba2f487fba9699aab"
server_key_version: "encryptionkey@0"
I20260925 22:43:56.512260 18063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:43:56.538472 18063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:43:56.542105 18063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:43:56.542325 18063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:43:56.542994 18063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:43:56.544018 18063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:43:56.544085 18063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:56.544150 18063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:43:56.544193 18063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:43:56.553823 18063 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:43753
I20260925 22:43:56.553841 18191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:43753 every 8 connection(s)
I20260925 22:43:56.554847 18063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:43:56.561851 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 18063
I20260925 22:43:56.562007 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance
I20260925 22:43:56.562299 2347 external_mini_cluster.cc:1428] Setting key f5ae2f82bd808ad8174e72266a059775
Sep 25 22:43:56 dist-test-slave-csnl krb5kdc[17688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376236, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:43:56.568877 17772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:56.556834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:36919 (local address 127.2.74.254:34049)
0925 22:43:56.556977 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:56.556981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:56.558041 (+ 1060us) server_negotiation.cc:407] Connection header received
0925 22:43:56.558995 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:56.558999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:56.559063 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:56.559167 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:56.560413 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.560958 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.562215 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.562492 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.562605 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:56.565577 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:56.565580 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:56.565582 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:56.565608 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:56.567122 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:56.567580 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:56.567583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:56.567584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:56.567658 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:56.567939 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:56.567941 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:56.567942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:56.568116 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:56.568190 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:56.568648 (+ 458us) server_negotiation.cc:299] Negotiation successful
0925 22:43:56.568754 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:43:56.569729 18194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:56.557189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.195:36919)
0925 22:43:56.557873 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:56.557916 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:56.558749 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:56.559349 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:56.559356 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:56.559766 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:43:56.560257 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:56.560270 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.561089 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.561093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:56.562063 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:56.562089 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.562658 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.562662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:56.562857 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:56.563559 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:43:56.563591 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:43:56.565421 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:43:56.567218 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:56.567223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:56.567236 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:56.567492 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:56.567730 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:43:56.567732 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:43:56.567733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:43:56.567866 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:43:56.568200 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:43:56.568205 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:43:56.568448 (+ 243us) client_negotiation.cc:769] Sending connection context
0925 22:43:56.568608 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 22:43:56.568870 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":126,"threads_started":1}
I20260925 22:43:56.570856 18192 heartbeater.cc:347] Connected to a master server at 127.2.74.254:34049
I20260925 22:43:56.571179 18192 heartbeater.cc:464] Registering TS with master...
I20260925 22:43:56.571699 18192 heartbeater.cc:511] Master 127.2.74.254:34049 requested a full tablet report, sending...
I20260925 22:43:56.572722 17733 ts_manager.cc:194] Registered new tserver with Master: c0a24c924c484ac2be5dbec2b5bb45e1 (127.2.74.195:43753)
I20260925 22:43:56.573289 17733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.195@KRBTEST.COM'} at 127.2.74.195:36919
I20260925 22:43:56.576572 2347 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:43:56.586059 17772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:56.578628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43368 (local address 127.2.74.254:34049)
0925 22:43:56.578771 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:56.578775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:56.579012 (+ 237us) server_negotiation.cc:407] Connection header received
0925 22:43:56.579125 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:56.579128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:56.579168 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:56.579255 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:43:56.579933 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.580451 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.581348 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.581591 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.581702 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:56.583260 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:43:56.583264 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:43:56.583266 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:43:56.583294 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:43:56.584545 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:56.584933 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:56.584935 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:56.584936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:56.585010 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:43:56.585227 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:43:56.585229 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:43:56.585230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:43:56.585446 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:43:56.585548 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:56.585759 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 22:43:56.585862 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:43:56.589154 17733 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:43:56.592445 17733 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:43:56.606406 18203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:56.600800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:43753 (local address 127.0.0.1:47746)
0925 22:43:56.601208 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:56.601266 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:56.601398 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:56.602150 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:56.602157 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:56.602181 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:43:56.602333 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:56.602344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.603597 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.603612 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:56.604601 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:56.604615 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.605789 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.605793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:56.605921 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:56.605961 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 22:43:56.606090 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 22:43:56.606244 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":104,"threads_started":1}
I20260925 22:43:56.606418 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:56.600971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47746 (local address 127.2.74.195:43753)
0925 22:43:56.601323 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:56.601336 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:56.601361 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:43:56.601565 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:56.601571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:56.601789 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:56.601944 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:43:56.602449 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.603476 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.604767 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.605669 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.605780 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:56.605930 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:56.606136 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 22:43:56.606263 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":127,"threads_started":1}
I20260925 22:43:56.607489 18207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:56.601942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:39625 (local address 127.0.0.1:53134)
0925 22:43:56.602562 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:56.602576 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:56.602652 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:56.603143 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:56.603147 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:56.603173 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:43:56.603276 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:56.603282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.604722 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.604726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:56.606019 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:56.606030 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.607080 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.607083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:56.607182 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:56.607204 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:43:56.607285 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 22:43:56.607360 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":77,"threads_started":1}
I20260925 22:43:56.607551 18208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:56.602338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53134 (local address 127.2.74.194:39625)
0925 22:43:56.602662 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:56.602677 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:56.602702 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:43:56.602779 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:56.602784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:56.603001 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:56.603147 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:43:56.603392 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.604557 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.606155 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.606923 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.607081 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:56.607231 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:56.607324 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 22:43:56.607415 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":108,"threads_started":1}
I20260925 22:43:56.607681 18205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:56.601590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:37037 (local address 127.0.0.1:48258)
0925 22:43:56.602275 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:56.602291 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:56.602369 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:56.603331 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:56.603352 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:56.603375 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:43:56.603470 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:56.603476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.604947 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.604950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:56.606100 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:56.606111 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.607304 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.607308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:56.607413 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:56.607436 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:43:56.607484 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 22:43:56.607550 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":100,"threads_started":1}
I20260925 22:43:56.608299 18206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:56.601802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48258 (local address 127.2.74.193:37037)
0925 22:43:56.602868 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:56.602874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:56.602894 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:43:56.602995 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:56.603000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:56.603176 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:56.603631 (+ 455us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:43:56.603770 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.604838 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.606251 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.607165 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.607868 (+ 703us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:56.608024 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:56.608074 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 22:43:56.608163 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":972,"thread_start_us":121,"threads_started":1}
I20260925 22:43:56.609998 18126 tablet_service.cc:1467] Processing CreateTablet for tablet 5ef7ece896504906a32d248211b70a69 (DEFAULT_TABLE table=test-table [id=749c04f172dd4905a4648b793bbe14ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:43:56.610443 17992 tablet_service.cc:1467] Processing CreateTablet for tablet 5ef7ece896504906a32d248211b70a69 (DEFAULT_TABLE table=test-table [id=749c04f172dd4905a4648b793bbe14ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:43:56.610939 18126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ef7ece896504906a32d248211b70a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:43:56.610690 17858 tablet_service.cc:1467] Processing CreateTablet for tablet 5ef7ece896504906a32d248211b70a69 (DEFAULT_TABLE table=test-table [id=749c04f172dd4905a4648b793bbe14ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:43:56.611511 17992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ef7ece896504906a32d248211b70a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:43:56.611526 17858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ef7ece896504906a32d248211b70a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:43:56.618193 18209 tablet_bootstrap.cc:492] T 5ef7ece896504906a32d248211b70a69 P c0a24c924c484ac2be5dbec2b5bb45e1: Bootstrap starting.
I20260925 22:43:56.619318 18211 tablet_bootstrap.cc:492] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd: Bootstrap starting.
I20260925 22:43:56.619832 18210 tablet_bootstrap.cc:492] T 5ef7ece896504906a32d248211b70a69 P 4bed4a0e54674a99ba4661ddb1ce6395: Bootstrap starting.
I20260925 22:43:56.621023 18209 tablet_bootstrap.cc:654] T 5ef7ece896504906a32d248211b70a69 P c0a24c924c484ac2be5dbec2b5bb45e1: Neither blocks nor log segments found. Creating new log.
I20260925 22:43:56.621814 18209 log.cc:824] T 5ef7ece896504906a32d248211b70a69 P c0a24c924c484ac2be5dbec2b5bb45e1: Log is configured to *not* fsync() on all Append() calls
I20260925 22:43:56.621960 18211 tablet_bootstrap.cc:654] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd: Neither blocks nor log segments found. Creating new log.
I20260925 22:43:56.622012 18210 tablet_bootstrap.cc:654] T 5ef7ece896504906a32d248211b70a69 P 4bed4a0e54674a99ba4661ddb1ce6395: Neither blocks nor log segments found. Creating new log.
I20260925 22:43:56.622835 18211 log.cc:824] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd: Log is configured to *not* fsync() on all Append() calls
I20260925 22:43:56.622835 18210 log.cc:824] T 5ef7ece896504906a32d248211b70a69 P 4bed4a0e54674a99ba4661ddb1ce6395: Log is configured to *not* fsync() on all Append() calls
I20260925 22:43:56.624141 18209 tablet_bootstrap.cc:492] T 5ef7ece896504906a32d248211b70a69 P c0a24c924c484ac2be5dbec2b5bb45e1: No bootstrap required, opened a new log
I20260925 22:43:56.624396 18209 ts_tablet_manager.cc:1397] T 5ef7ece896504906a32d248211b70a69 P c0a24c924c484ac2be5dbec2b5bb45e1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:43:56.624979 18211 tablet_bootstrap.cc:492] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd: No bootstrap required, opened a new log
I20260925 22:43:56.625155 18210 tablet_bootstrap.cc:492] T 5ef7ece896504906a32d248211b70a69 P 4bed4a0e54674a99ba4661ddb1ce6395: No bootstrap required, opened a new log
I20260925 22:43:56.625185 18211 ts_tablet_manager.cc:1397] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260925 22:43:56.625348 18210 ts_tablet_manager.cc:1397] T 5ef7ece896504906a32d248211b70a69 P 4bed4a0e54674a99ba4661ddb1ce6395: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260925 22:43:56.628314 18211 raft_consensus.cc:348] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 } } peers { permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 } } peers { permanent_uuid: "c767841753fa44429e8c8185fa5114fd" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39625 } }
I20260925 22:43:56.628598 18211 raft_consensus.cc:374] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:43:56.628654 18211 raft_consensus.cc:729] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c767841753fa44429e8c8185fa5114fd, State: Initialized, Role: FOLLOWER
I20260925 22:43:56.628691 18209 raft_consensus.cc:348] T 5ef7ece896504906a32d248211b70a69 P c0a24c924c484ac2be5dbec2b5bb45e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 } } peers { permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 } } peers { permanent_uuid: "c767841753fa44429e8c8185fa5114fd" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39625 } }
I20260925 22:43:56.628952 18209 raft_consensus.cc:374] T 5ef7ece896504906a32d248211b70a69 P c0a24c924c484ac2be5dbec2b5bb45e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:43:56.629015 18209 raft_consensus.cc:729] T 5ef7ece896504906a32d248211b70a69 P c0a24c924c484ac2be5dbec2b5bb45e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0a24c924c484ac2be5dbec2b5bb45e1, State: Initialized, Role: FOLLOWER
I20260925 22:43:56.629186 18211 consensus_queue.cc:260] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 } } peers { permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 } } peers { permanent_uuid: "c767841753fa44429e8c8185fa5114fd" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39625 } }
I20260925 22:43:56.629462 18209 consensus_queue.cc:260] T 5ef7ece896504906a32d248211b70a69 P c0a24c924c484ac2be5dbec2b5bb45e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 } } peers { permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 } } peers { permanent_uuid: "c767841753fa44429e8c8185fa5114fd" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39625 } }
I20260925 22:43:56.629959 18058 heartbeater.cc:503] Master 127.2.74.254:34049 was elected leader, sending a full tablet report...
I20260925 22:43:56.629882 18210 raft_consensus.cc:348] T 5ef7ece896504906a32d248211b70a69 P 4bed4a0e54674a99ba4661ddb1ce6395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 } } peers { permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 } } peers { permanent_uuid: "c767841753fa44429e8c8185fa5114fd" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39625 } }
I20260925 22:43:56.630111 18210 raft_consensus.cc:374] T 5ef7ece896504906a32d248211b70a69 P 4bed4a0e54674a99ba4661ddb1ce6395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:43:56.630156 18210 raft_consensus.cc:729] T 5ef7ece896504906a32d248211b70a69 P 4bed4a0e54674a99ba4661ddb1ce6395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bed4a0e54674a99ba4661ddb1ce6395, State: Initialized, Role: FOLLOWER
I20260925 22:43:56.630342 18211 ts_tablet_manager.cc:1428] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:43:56.630541 18209 ts_tablet_manager.cc:1428] T 5ef7ece896504906a32d248211b70a69 P c0a24c924c484ac2be5dbec2b5bb45e1: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:43:56.630556 18210 consensus_queue.cc:260] T 5ef7ece896504906a32d248211b70a69 P 4bed4a0e54674a99ba4661ddb1ce6395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 } } peers { permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 } } peers { permanent_uuid: "c767841753fa44429e8c8185fa5114fd" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39625 } }
I20260925 22:43:56.630925 18192 heartbeater.cc:503] Master 127.2.74.254:34049 was elected leader, sending a full tablet report...
I20260925 22:43:56.631543 18210 ts_tablet_manager.cc:1428] T 5ef7ece896504906a32d248211b70a69 P 4bed4a0e54674a99ba4661ddb1ce6395: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260925 22:43:56.631589 17924 heartbeater.cc:503] Master 127.2.74.254:34049 was elected leader, sending a full tablet report...
I20260925 22:43:56.632158 18218 raft_consensus.cc:482] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:43:56.632289 18218 raft_consensus.cc:504] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 } } peers { permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 } } peers { permanent_uuid: "c767841753fa44429e8c8185fa5114fd" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39625 } }
I20260925 22:43:56.633536 18218 leader_election.cc:288] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0a24c924c484ac2be5dbec2b5bb45e1 (127.2.74.195:43753), 4bed4a0e54674a99ba4661ddb1ce6395 (127.2.74.193:37037)
I20260925 22:43:56.638330 18060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:56.633804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:43753 (local address 127.2.74.194:34161)
0925 22:43:56.634148 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:56.634162 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:56.634275 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:56.634613 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:56.634616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:56.634634 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:43:56.634763 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:56.634768 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.635716 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.635721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:56.636916 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:56.636927 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.637905 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.637908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:56.638014 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:56.638037 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:43:56.638107 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 22:43:56.638192 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267}
I20260925 22:43:56.638713 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:56.633901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:34161 (local address 127.2.74.195:43753)
0925 22:43:56.634021 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:56.634025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:56.634231 (+ 206us) server_negotiation.cc:407] Connection header received
0925 22:43:56.634460 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:56.634464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:56.634518 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:56.634600 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:43:56.634877 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.635585 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.637057 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.637679 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.637799 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:56.637849 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:56.638508 (+ 659us) server_negotiation.cc:299] Negotiation successful
0925 22:43:56.638589 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:43:56.639407 18219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:56.634081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:37037 (local address 127.2.74.194:46921)
0925 22:43:56.635524 (+ 1443us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:56.635537 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:43:56.635651 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:43:56.635962 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:43:56.635965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:43:56.635978 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:43:56.636086 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:56.636092 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.636845 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.636847 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:56.638029 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:43:56.638037 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.638999 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.639003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:43:56.639107 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:56.639125 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 22:43:56.639214 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 22:43:56.639284 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1381,"thread_start_us":113,"threads_started":1}
I20260925 22:43:56.639354 18146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ef7ece896504906a32d248211b70a69" candidate_uuid: "c767841753fa44429e8c8185fa5114fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" is_pre_election: true
I20260925 22:43:56.639425 18206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:56.634532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:46921 (local address 127.2.74.193:37037)
0925 22:43:56.634648 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:56.634653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:56.635617 (+ 964us) server_negotiation.cc:407] Connection header received
0925 22:43:56.635740 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:56.635746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:56.635794 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:56.635861 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:43:56.636228 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.636729 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.638161 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.638788 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.638899 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:56.639000 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:56.639244 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 22:43:56.639319 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 22:43:56.639724 18146 raft_consensus.cc:2454] T 5ef7ece896504906a32d248211b70a69 P c0a24c924c484ac2be5dbec2b5bb45e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c767841753fa44429e8c8185fa5114fd in term 0.
I20260925 22:43:56.639978 17878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ef7ece896504906a32d248211b70a69" candidate_uuid: "c767841753fa44429e8c8185fa5114fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" is_pre_election: true
I20260925 22:43:56.640226 17947 leader_election.cc:302] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0a24c924c484ac2be5dbec2b5bb45e1, c767841753fa44429e8c8185fa5114fd; no voters:
I20260925 22:43:56.640260 17878 raft_consensus.cc:2454] T 5ef7ece896504906a32d248211b70a69 P 4bed4a0e54674a99ba4661ddb1ce6395 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c767841753fa44429e8c8185fa5114fd in term 0.
I20260925 22:43:56.640509 18218 raft_consensus.cc:2781] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:43:56.640604 18218 raft_consensus.cc:482] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:43:56.640663 18218 raft_consensus.cc:3037] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:43:56.641775 18218 raft_consensus.cc:504] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 } } peers { permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 } } peers { permanent_uuid: "c767841753fa44429e8c8185fa5114fd" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39625 } }
I20260925 22:43:56.642181 18218 leader_election.cc:288] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd [CANDIDATE]: Term 1 election: Requested vote from peers c0a24c924c484ac2be5dbec2b5bb45e1 (127.2.74.195:43753), 4bed4a0e54674a99ba4661ddb1ce6395 (127.2.74.193:37037)
I20260925 22:43:56.642652 17878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ef7ece896504906a32d248211b70a69" candidate_uuid: "c767841753fa44429e8c8185fa5114fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bed4a0e54674a99ba4661ddb1ce6395"
I20260925 22:43:56.642768 17878 raft_consensus.cc:3037] T 5ef7ece896504906a32d248211b70a69 P 4bed4a0e54674a99ba4661ddb1ce6395 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:43:56.643344 18146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ef7ece896504906a32d248211b70a69" candidate_uuid: "c767841753fa44429e8c8185fa5114fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1"
I20260925 22:43:56.643491 18146 raft_consensus.cc:3037] T 5ef7ece896504906a32d248211b70a69 P c0a24c924c484ac2be5dbec2b5bb45e1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:43:56.644109 17878 raft_consensus.cc:2454] T 5ef7ece896504906a32d248211b70a69 P 4bed4a0e54674a99ba4661ddb1ce6395 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c767841753fa44429e8c8185fa5114fd in term 1.
I20260925 22:43:56.644472 17947 leader_election.cc:302] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bed4a0e54674a99ba4661ddb1ce6395, c767841753fa44429e8c8185fa5114fd; no voters:
I20260925 22:43:56.644686 18218 raft_consensus.cc:2781] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:43:56.644825 18218 raft_consensus.cc:686] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd [term 1 LEADER]: Becoming Leader. State: Replica: c767841753fa44429e8c8185fa5114fd, State: Running, Role: LEADER
I20260925 22:43:56.644865 18146 raft_consensus.cc:2454] T 5ef7ece896504906a32d248211b70a69 P c0a24c924c484ac2be5dbec2b5bb45e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c767841753fa44429e8c8185fa5114fd in term 1.
I20260925 22:43:56.645229 18218 consensus_queue.cc:237] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 } } peers { permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 } } peers { permanent_uuid: "c767841753fa44429e8c8185fa5114fd" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39625 } }
I20260925 22:43:56.648800 17733 catalog_manager.cc:5495] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c767841753fa44429e8c8185fa5114fd (127.2.74.194). New cstate: current_term: 1 leader_uuid: "c767841753fa44429e8c8185fa5114fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c767841753fa44429e8c8185fa5114fd" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39625 } health_report { overall_health: HEALTHY } } }
I20260925 22:43:56.662225 18208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:56.658692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53136 (local address 127.2.74.194:39625)
0925 22:43:56.658835 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:43:56.658838 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:43:56.658851 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:43:56.658926 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:43:56.658929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:43:56.659059 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:43:56.659160 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:43:56.659399 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.659905 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.660805 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:43:56.661028 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:43:56.661114 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:43:56.661405 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:43:56.661840 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:43:56.661935 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 22:43:56.662045 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 22:43:56.662102 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:43:56.668465 18146 raft_consensus.cc:1261] T 5ef7ece896504906a32d248211b70a69 P c0a24c924c484ac2be5dbec2b5bb45e1 [term 1 FOLLOWER]: Refusing update from remote peer c767841753fa44429e8c8185fa5114fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:43:56.668465 17878 raft_consensus.cc:1261] T 5ef7ece896504906a32d248211b70a69 P 4bed4a0e54674a99ba4661ddb1ce6395 [term 1 FOLLOWER]: Refusing update from remote peer c767841753fa44429e8c8185fa5114fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:43:56.669224 18218 consensus_queue.cc:1035] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:43:56.669418 18215 consensus_queue.cc:1035] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:43:56.677899 18224 mvcc.cc:205] Tried to move back new op lower bound from 7333381065386192896 to 7333381065301897216. Current Snapshot: MvccSnapshot[applied={T|T < 7333381065386192896}]
I20260925 22:43:56.678140 18223 mvcc.cc:205] Tried to move back new op lower bound from 7333381065386192896 to 7333381065301897216. Current Snapshot: MvccSnapshot[applied={T|T < 7333381065386192896}]
I20260925 22:43:56.686903 17732 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:43:56.687134 17732 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:43:56.689837 17732 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9: Sending DeleteTablet for 3 replicas of tablet 5ef7ece896504906a32d248211b70a69
I20260925 22:43:56.690675 18126 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ef7ece896504906a32d248211b70a69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:43:56 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:47746
I20260925 22:43:56.690791 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 17795
I20260925 22:43:56.690790 17858 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ef7ece896504906a32d248211b70a69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:43:56 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:48258
I20260925 22:43:56.690929 17992 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ef7ece896504906a32d248211b70a69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:43:56 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:53134
I20260925 22:43:56.692821 18233 tablet_replica.cc:330] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd: stopping tablet replica
I20260925 22:43:56.693248 18233 raft_consensus.cc:2229] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:43:56.693718 18233 raft_consensus.cc:2258] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:43:56.695537 18233 ts_tablet_manager.cc:1905] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 22:43:56.698661 17719 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.74.193:37037 (error 108)
I20260925 22:43:56.698727 18232 tablet_replica.cc:330] T 5ef7ece896504906a32d248211b70a69 P c0a24c924c484ac2be5dbec2b5bb45e1: stopping tablet replica
I20260925 22:43:56.699100 18233 ts_tablet_manager.cc:1918] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:43:56.699214 18233 log.cc:1196] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/wal/wals/5ef7ece896504906a32d248211b70a69
I20260925 22:43:56.699391 18232 raft_consensus.cc:2229] T 5ef7ece896504906a32d248211b70a69 P c0a24c924c484ac2be5dbec2b5bb45e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:43:56.699571 18233 ts_tablet_manager.cc:1939] T 5ef7ece896504906a32d248211b70a69 P c767841753fa44429e8c8185fa5114fd: Deleting consensus metadata
I20260925 22:43:56.699883 18232 raft_consensus.cc:2258] T 5ef7ece896504906a32d248211b70a69 P c0a24c924c484ac2be5dbec2b5bb45e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:43:56.700342 18205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:56.699969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:37037 (local address 127.0.0.1:48268)
0925 22:43:56.700173 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:56.700244 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.193:37037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260925 22:43:56.700522 17719 catalog_manager.cc:4584] TS 4bed4a0e54674a99ba4661ddb1ce6395 (127.2.74.193:37037): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5ef7ece896504906a32d248211b70a69: Network error: Client connection negotiation failed: client connection to 127.2.74.193:37037: connect: Connection refused (error 111)
I20260925 22:43:56.700858 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 17929
I20260925 22:43:56.702231 18232 ts_tablet_manager.cc:1905] T 5ef7ece896504906a32d248211b70a69 P c0a24c924c484ac2be5dbec2b5bb45e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:43:56.705070 18232 ts_tablet_manager.cc:1918] T 5ef7ece896504906a32d248211b70a69 P c0a24c924c484ac2be5dbec2b5bb45e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:43:56.705185 18232 log.cc:1196] T 5ef7ece896504906a32d248211b70a69 P c0a24c924c484ac2be5dbec2b5bb45e1: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/wal/wals/5ef7ece896504906a32d248211b70a69
I20260925 22:43:56.705498 18232 ts_tablet_manager.cc:1939] T 5ef7ece896504906a32d248211b70a69 P c0a24c924c484ac2be5dbec2b5bb45e1: Deleting consensus metadata
I20260925 22:43:56.707031 17719 catalog_manager.cc:4841] TS c0a24c924c484ac2be5dbec2b5bb45e1 (127.2.74.195:43753): tablet 5ef7ece896504906a32d248211b70a69 (table test-table [id=749c04f172dd4905a4648b793bbe14ff]) successfully deleted
I20260925 22:43:56.708360 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 18063
I20260925 22:43:56.708681 18205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:43:56.708309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:39625 (local address 127.0.0.1:53152)
0925 22:43:56.708497 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 22:43:56.708570 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.194:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 22:43:56.714555 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 17701
I20260925 22:44:16.721740 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:34049
--webserver_interface=127.2.74.254
--webserver_port=43585
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:43275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:34049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:16.840047 18235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:16.840516 18235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:16.840607 18235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:16.846021 18235 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 22:44:16.846109 18235 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 22:44:16.846127 18235 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 22:44:16.846143 18235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:44:16.846161 18235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:16.846180 18235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:16.846196 18235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:44:16.846213 18235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:44:16.852238 18235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:43275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:34049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:34049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=43585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:44:16.853380 18235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:16.854434 18235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:16.868531 18244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:16.868531 18242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:16.868531 18241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:16.868923 18235 server_base.cc:1034] running on GCE node
I20260925 22:44:16.869645 18235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:16.871088 18235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:16.872320 18235 hybrid_clock.cc:648] HybridClock initialized: now 1790376256872309 us; error 41 us; skew 500 ppm
Sep 25 22:44:16 dist-test-slave-csnl krb5kdc[17688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376256, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:16.879240 18235 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:44:16.880417 18235 webserver.cc:460] Webserver started at http://127.2.74.254:43585/ using document root <none> and password file <none>
I20260925 22:44:16.881129 18235 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:16.881266 18235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:16.885725 18235 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:44:16.888182 18251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:16.889648 18235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:44:16.889835 18235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "0e78555668e843efbb9de814ef9dcbe9"
format_stamp: "Formatted at 2026-09-25 22:43:55 on dist-test-slave-csnl"
server_key: "e63f0d422b6e8cce8d9c5b6516d9fddd"
server_key_iv: "663e048941c2f3de57ed20ebb28df4a4"
server_key_version: "encryptionkey@0"
I20260925 22:44:16.890532 18235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:16.905019 18235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:16.908588 18235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:16.908880 18235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:16.916723 18235 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:34049
I20260925 22:44:16.917028 18303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:34049 every 8 connection(s)
I20260925 22:44:16.918009 18235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:44:16.919248 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 18235
I20260925 22:44:16.919691 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:37037
--local_ip_for_outbound_sockets=127.2.74.193
--tserver_master_addrs=127.2.74.254:34049
--webserver_port=44489
--webserver_interface=127.2.74.193
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:43275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:44:16.923995 18304 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:44:16.926939 18304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9: Bootstrap starting.
I20260925 22:44:16.950676 18304 log.cc:824] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:16.961884 18304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:44:16.962393 18304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9: Bootstrap complete.
I20260925 22:44:16.968211 18304 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e78555668e843efbb9de814ef9dcbe9" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34049 } }
I20260925 22:44:16.968789 18304 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e78555668e843efbb9de814ef9dcbe9, State: Initialized, Role: FOLLOWER
I20260925 22:44:16.969358 18304 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e78555668e843efbb9de814ef9dcbe9" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34049 } }
I20260925 22:44:16.969515 18304 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:44:16.969588 18304 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:44:16.969696 18304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:44:16.972446 18304 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e78555668e843efbb9de814ef9dcbe9" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34049 } }
I20260925 22:44:16.972852 18304 leader_election.cc:302] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e78555668e843efbb9de814ef9dcbe9; no voters:
I20260925 22:44:16.973222 18304 leader_election.cc:288] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 22:44:16.973399 18308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 22:44:16.973728 18308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [term 2 LEADER]: Becoming Leader. State: Replica: 0e78555668e843efbb9de814ef9dcbe9, State: Running, Role: LEADER
I20260925 22:44:16.974072 18308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e78555668e843efbb9de814ef9dcbe9" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34049 } }
I20260925 22:44:16.974555 18304 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:44:16.975526 18310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e78555668e843efbb9de814ef9dcbe9. Latest consensus state: current_term: 2 leader_uuid: "0e78555668e843efbb9de814ef9dcbe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e78555668e843efbb9de814ef9dcbe9" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34049 } } }
I20260925 22:44:16.975677 18310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:16.976011 18309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0e78555668e843efbb9de814ef9dcbe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e78555668e843efbb9de814ef9dcbe9" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34049 } } }
I20260925 22:44:16.976106 18309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:16.976729 18317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:44:16.982266 18317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:44:16.983150 18317 catalog_manager.cc:1259] Loaded cluster ID: b95bcf3b1ddc4e09a3b9423c1f0db741
I20260925 22:44:16.983237 18317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:44:16.985950 18317 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:44:16.986896 18317 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9: Loaded TSK: 0
I20260925 22:44:17.005049 18317 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9: Generated new TSK 1
I20260925 22:44:17.005384 18317 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:44:17.062322 18306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:17.062635 18306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:17.062683 18306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:17.067252 18306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:17.067332 18306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:17.067432 18306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:44:17.073278 18306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:43275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:37037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=44489
--tserver_master_addrs=127.2.74.254:34049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:44:17.074546 18306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:17.075875 18306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:17.086735 18332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:17.086831 18331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:17.086760 18334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:17.087252 18306 server_base.cc:1034] running on GCE node
I20260925 22:44:17.087705 18306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:17.088428 18306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:17.089614 18306 hybrid_clock.cc:648] HybridClock initialized: now 1790376257089580 us; error 58 us; skew 500 ppm
Sep 25 22:44:17 dist-test-slave-csnl krb5kdc[17688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376257, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:17.092900 18306 init.cc:377] Logged in from keytab as kudu/127.2.74.193@KRBTEST.COM (short username kudu)
I20260925 22:44:17.093976 18306 webserver.cc:460] Webserver started at http://127.2.74.193:44489/ using document root <none> and password file <none>
I20260925 22:44:17.094705 18306 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:17.094827 18306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:17.099565 18306 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:17.102260 18341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:17.103842 18306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:17.104013 18306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "4bed4a0e54674a99ba4661ddb1ce6395"
format_stamp: "Formatted at 2026-09-25 22:43:56 on dist-test-slave-csnl"
server_key: "62831131f6d0b5de48446fa4ad439ff8"
server_key_iv: "f5dbf9d44f655741b1882dbcd3840308"
server_key_version: "encryptionkey@0"
I20260925 22:44:17.104640 18306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:17.126506 18306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:17.130172 18306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:17.130400 18306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:17.131125 18306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:17.132300 18348 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 22:44:17.137863 18306 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 22:44:17.137960 18306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260925 22:44:17.138034 18306 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 22:44:17.140528 18306 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 22:44:17.140630 18306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.001s
I20260925 22:44:17.141029 18348 tablet_bootstrap.cc:492] T 5ef7ece896504906a32d248211b70a69 P 4bed4a0e54674a99ba4661ddb1ce6395: Bootstrap starting.
I20260925 22:44:17.151741 18306 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:37037
I20260925 22:44:17.151850 18455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:37037 every 8 connection(s)
I20260925 22:44:17.152889 18306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:44:17.158290 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 18306
I20260925 22:44:17.158805 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:39625
--local_ip_for_outbound_sockets=127.2.74.194
--tserver_master_addrs=127.2.74.254:34049
--webserver_port=33473
--webserver_interface=127.2.74.194
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:43275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:44:17.163298 18348 log.cc:824] T 5ef7ece896504906a32d248211b70a69 P 4bed4a0e54674a99ba4661ddb1ce6395: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:44:17 dist-test-slave-csnl krb5kdc[17688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376257, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:44:17.173137 18348 tablet_bootstrap.cc:492] T 5ef7ece896504906a32d248211b70a69 P 4bed4a0e54674a99ba4661ddb1ce6395: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:44:17.173621 18348 tablet_bootstrap.cc:492] T 5ef7ece896504906a32d248211b70a69 P 4bed4a0e54674a99ba4661ddb1ce6395: Bootstrap complete.
I20260925 22:44:17.175102 18348 ts_tablet_manager.cc:1397] T 5ef7ece896504906a32d248211b70a69 P 4bed4a0e54674a99ba4661ddb1ce6395: Time spent bootstrapping tablet: real 0.034s user 0.022s sys 0.012s
I20260925 22:44:17.175329 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:17.155542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.193:43733)
0925 22:44:17.156220 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:17.156319 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:17.157539 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:17.159818 (+ 2279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:17.159832 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:17.160442 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:17.161213 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:17.161237 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.162681 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.162685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:17.163840 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:17.163854 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.165121 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.165124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:17.165269 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:17.165935 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:17.165952 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:17.167947 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:17.171251 (+ 3304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:17.171257 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:17.171271 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:17.171638 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:17.172141 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:17.172144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:17.172146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:17.172319 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:17.173132 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:17.173138 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:17.173453 (+ 315us) client_negotiation.cc:769] Sending connection context
0925 22:44:17.173676 (+ 223us) client_negotiation.cc:240] Negotiation successful
0925 22:44:17.174072 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":175,"threads_started":1}
I20260925 22:44:17.175657 18458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:17.155539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:43733 (local address 127.2.74.254:34049)
0925 22:44:17.155973 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:17.155994 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:17.156481 (+ 487us) server_negotiation.cc:407] Connection header received
0925 22:44:17.158161 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:17.158180 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:17.158690 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:17.159230 (+ 540us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:17.161449 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.162494 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.164014 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.164783 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.164899 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:17.168279 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:17.168290 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:17.168318 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:17.168366 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:17.171066 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:17.171797 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:17.171807 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:17.171815 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:17.172012 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:17.172441 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:17.172448 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:17.172452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:17.172996 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:17.173134 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:17.173877 (+ 743us) server_negotiation.cc:299] Negotiation successful
0925 22:44:17.174213 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":133,"threads_started":1}
I20260925 22:44:17.178634 18456 heartbeater.cc:347] Connected to a master server at 127.2.74.254:34049
I20260925 22:44:17.178921 18456 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:17.178542 18348 raft_consensus.cc:348] T 5ef7ece896504906a32d248211b70a69 P 4bed4a0e54674a99ba4661ddb1ce6395 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 } } peers { permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 } } peers { permanent_uuid: "c767841753fa44429e8c8185fa5114fd" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39625 } }
I20260925 22:44:17.179102 18348 raft_consensus.cc:729] T 5ef7ece896504906a32d248211b70a69 P 4bed4a0e54674a99ba4661ddb1ce6395 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bed4a0e54674a99ba4661ddb1ce6395, State: Initialized, Role: FOLLOWER
I20260925 22:44:17.179582 18348 consensus_queue.cc:260] T 5ef7ece896504906a32d248211b70a69 P 4bed4a0e54674a99ba4661ddb1ce6395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 } } peers { permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 } } peers { permanent_uuid: "c767841753fa44429e8c8185fa5114fd" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39625 } }
I20260925 22:44:17.179589 18456 heartbeater.cc:511] Master 127.2.74.254:34049 requested a full tablet report, sending...
I20260925 22:44:17.180647 18348 ts_tablet_manager.cc:1428] T 5ef7ece896504906a32d248211b70a69 P 4bed4a0e54674a99ba4661ddb1ce6395: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:44:17.182624 18268 ts_manager.cc:194] Registered new tserver with Master: 4bed4a0e54674a99ba4661ddb1ce6395 (127.2.74.193:37037)
I20260925 22:44:17.186820 18268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.193@KRBTEST.COM'} at 127.2.74.193:43733
I20260925 22:44:17.189916 18456 heartbeater.cc:503] Master 127.2.74.254:34049 was elected leader, sending a full tablet report...
Sep 25 22:44:17 dist-test-slave-csnl krb5kdc[17688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376256, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for kudu/127.2.74.193@KRBTEST.COM
I20260925 22:44:17.200240 18463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:17.185892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:37037 (local address 127.0.0.1:37856)
0925 22:44:17.186348 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:17.186402 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:17.186562 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:17.187323 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:17.187329 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:17.187841 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:17.187983 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:17.187992 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.191405 (+ 3413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.191410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:17.192430 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:17.192444 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.193117 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.193121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:17.193257 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:17.193935 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:17.193959 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:17.195937 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:17.198179 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:17.198192 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:17.198198 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:17.198467 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:17.198819 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:17.198822 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:17.198824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:17.198921 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:17.199477 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:17.199484 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:17.199629 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 22:44:17.199791 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 22:44:17.200037 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":111,"threads_started":1}
I20260925 22:44:17.200240 18464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:17.185986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37856 (local address 127.2.74.193:37037)
0925 22:44:17.186744 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:17.186756 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:17.186786 (+ 30us) server_negotiation.cc:407] Connection header received
0925 22:44:17.186868 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:17.186873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:17.187148 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:17.187298 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:17.190195 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.191250 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.192599 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.192976 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.193079 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:17.196199 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:17.196211 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:17.196223 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:17.196250 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:17.197989 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:17.198617 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:17.198624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:17.198625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:17.198712 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:17.199053 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:17.199059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:17.199063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:17.199351 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:17.199503 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:17.199877 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 22:44:17.200058 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":158,"threads_started":1}
I20260925 22:44:17.201335 18381 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ef7ece896504906a32d248211b70a69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:43:56 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:37856
I20260925 22:44:17.202276 18465 tablet_replica.cc:330] stopping tablet replica
I20260925 22:44:17.202632 18465 raft_consensus.cc:2229] T 5ef7ece896504906a32d248211b70a69 P 4bed4a0e54674a99ba4661ddb1ce6395 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:17.202924 18465 raft_consensus.cc:2258] T 5ef7ece896504906a32d248211b70a69 P 4bed4a0e54674a99ba4661ddb1ce6395 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:17.204183 18465 ts_tablet_manager.cc:1905] T 5ef7ece896504906a32d248211b70a69 P 4bed4a0e54674a99ba4661ddb1ce6395: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:17.209544 18465 ts_tablet_manager.cc:1918] T 5ef7ece896504906a32d248211b70a69 P 4bed4a0e54674a99ba4661ddb1ce6395: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:44:17.209671 18465 log.cc:1196] T 5ef7ece896504906a32d248211b70a69 P 4bed4a0e54674a99ba4661ddb1ce6395: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/wal/wals/5ef7ece896504906a32d248211b70a69
I20260925 22:44:17.209959 18465 ts_tablet_manager.cc:1939] T 5ef7ece896504906a32d248211b70a69 P 4bed4a0e54674a99ba4661ddb1ce6395: Deleting consensus metadata
I20260925 22:44:17.212209 18254 catalog_manager.cc:4841] TS 4bed4a0e54674a99ba4661ddb1ce6395 (127.2.74.193:37037): tablet 5ef7ece896504906a32d248211b70a69 (table test-table [id=749c04f172dd4905a4648b793bbe14ff]) successfully deleted
W20260925 22:44:17.289522 18460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:17.289827 18460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:17.289873 18460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:17.294240 18460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:17.294317 18460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:17.294379 18460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:44:17.300194 18460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:43275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:39625
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=33473
--tserver_master_addrs=127.2.74.254:34049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:44:17.301368 18460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:17.302456 18460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:17.313071 18471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:17.313081 18473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:17.313071 18470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:17.313380 18460 server_base.cc:1034] running on GCE node
I20260925 22:44:17.314087 18460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:17.314787 18460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:17.315950 18460 hybrid_clock.cc:648] HybridClock initialized: now 1790376257315930 us; error 34 us; skew 500 ppm
Sep 25 22:44:17 dist-test-slave-csnl krb5kdc[17688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376257, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:17.318810 18460 init.cc:377] Logged in from keytab as kudu/127.2.74.194@KRBTEST.COM (short username kudu)
I20260925 22:44:17.319845 18460 webserver.cc:460] Webserver started at http://127.2.74.194:33473/ using document root <none> and password file <none>
I20260925 22:44:17.320449 18460 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:17.320554 18460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:17.324780 18460 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 22:44:17.327090 18480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:17.328528 18460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:44:17.328709 18460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "c767841753fa44429e8c8185fa5114fd"
format_stamp: "Formatted at 2026-09-25 22:43:56 on dist-test-slave-csnl"
server_key: "bf3482e6a04f9d506d0dfede7a345283"
server_key_iv: "51fbcdda3594314272adb636f7511411"
server_key_version: "encryptionkey@0"
I20260925 22:44:17.329339 18460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:17.349941 18460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:17.353322 18460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:17.353572 18460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:17.354242 18460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:17.355276 18460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:17.355355 18460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:17.355435 18460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:17.355481 18460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:17.365374 18460 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:39625
I20260925 22:44:17.365396 18593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:39625 every 8 connection(s)
I20260925 22:44:17.366573 18460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:44:17.368376 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 18460
I20260925 22:44:17.368935 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:43753
--local_ip_for_outbound_sockets=127.2.74.195
--tserver_master_addrs=127.2.74.254:34049
--webserver_port=36811
--webserver_interface=127.2.74.195
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:43275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:44:17 dist-test-slave-csnl krb5kdc[17688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376257, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:44:17.387223 18458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:17.368871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:42009 (local address 127.2.74.254:34049)
0925 22:44:17.369076 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:17.369083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:17.370557 (+ 1474us) server_negotiation.cc:407] Connection header received
0925 22:44:17.371619 (+ 1062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:17.371626 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:17.371708 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:17.371820 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:17.373340 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.374383 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.376261 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.376748 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.376970 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:17.381660 (+ 4690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:17.381666 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:17.381668 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:17.381733 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:17.383986 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:17.384852 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:17.384856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:17.384858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:17.384977 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:17.385460 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:17.385463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:17.385464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:17.385691 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:17.385783 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:17.386684 (+ 901us) server_negotiation.cc:299] Negotiation successful
0925 22:44:17.386914 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 22:44:17.388178 18596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:17.369423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.194:42009)
0925 22:44:17.370331 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:17.370402 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:17.371365 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:17.372010 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:17.372046 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:17.372460 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:17.373122 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:17.373142 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.374571 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.374574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:17.376062 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:17.376079 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.377298 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.377305 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:17.377626 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:17.378710 (+ 1084us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:17.378750 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:17.381292 (+ 2542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:17.384177 (+ 2885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:17.384185 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:17.384214 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:17.384668 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:17.385084 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:17.385087 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:17.385089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:17.385311 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:17.385803 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:17.385811 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:17.386215 (+ 404us) client_negotiation.cc:769] Sending connection context
0925 22:44:17.386469 (+ 254us) client_negotiation.cc:240] Negotiation successful
0925 22:44:17.386988 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":137,"threads_started":1}
I20260925 22:44:17.389848 18594 heartbeater.cc:347] Connected to a master server at 127.2.74.254:34049
I20260925 22:44:17.390220 18594 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:17.390856 18594 heartbeater.cc:511] Master 127.2.74.254:34049 requested a full tablet report, sending...
I20260925 22:44:17.392320 18268 ts_manager.cc:194] Registered new tserver with Master: c767841753fa44429e8c8185fa5114fd (127.2.74.194:39625)
I20260925 22:44:17.393146 18268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.194@KRBTEST.COM'} at 127.2.74.194:42009
W20260925 22:44:17.494633 18597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:17.495039 18597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:17.495122 18597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:17.499610 18597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:17.499709 18597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:17.499776 18597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:44:17.506160 18597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:43275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:43753
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=36811
--tserver_master_addrs=127.2.74.254:34049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:44:17.507434 18597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:17.508554 18597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:17.519129 18602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:17.519179 18605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:17.519166 18603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:17.519703 18597 server_base.cc:1034] running on GCE node
I20260925 22:44:17.520216 18597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:17.520942 18597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:17.522159 18597 hybrid_clock.cc:648] HybridClock initialized: now 1790376257522115 us; error 59 us; skew 500 ppm
Sep 25 22:44:17 dist-test-slave-csnl krb5kdc[17688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376257, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:17.525339 18597 init.cc:377] Logged in from keytab as kudu/127.2.74.195@KRBTEST.COM (short username kudu)
I20260925 22:44:17.526381 18597 webserver.cc:460] Webserver started at http://127.2.74.195:36811/ using document root <none> and password file <none>
I20260925 22:44:17.527045 18597 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:17.527158 18597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:17.531342 18597 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:44:17.533751 18612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:17.535300 18597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:17.535430 18597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "c0a24c924c484ac2be5dbec2b5bb45e1"
format_stamp: "Formatted at 2026-09-25 22:43:56 on dist-test-slave-csnl"
server_key: "df8405a897aaa0f23d64580c402fbd5f"
server_key_iv: "ec828e616be0374ba2f487fba9699aab"
server_key_version: "encryptionkey@0"
I20260925 22:44:17.536044 18597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:17.553874 18597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:17.557482 18597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:17.557662 18597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:17.558247 18597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:17.559124 18597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:17.559173 18597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:17.559216 18597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:17.559232 18597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:17.568312 18597 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:43753
I20260925 22:44:17.568336 18725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:43753 every 8 connection(s)
I20260925 22:44:17.569404 18597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:44:17.576864 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 18597
Sep 25 22:44:17 dist-test-slave-csnl krb5kdc[17688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376257, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:44:17.584694 18458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:17.571522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:46779 (local address 127.2.74.254:34049)
0925 22:44:17.571676 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:17.571680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:17.572708 (+ 1028us) server_negotiation.cc:407] Connection header received
0925 22:44:17.573661 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:17.573665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:17.573726 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:17.573836 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:17.575237 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.575845 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.577269 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.577535 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.577658 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:17.580488 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:17.580493 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:17.580496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:17.580524 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:17.582268 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:17.582983 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:17.582986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:17.582988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:17.583083 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:17.583478 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:17.583488 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:17.583489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:17.583668 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:17.583779 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:17.584313 (+ 534us) server_negotiation.cc:299] Negotiation successful
0925 22:44:17.584508 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:44:17.585609 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:17.571854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.195:46779)
0925 22:44:17.572550 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:17.572593 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:17.573441 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:17.574028 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:17.574036 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:17.574423 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:17.575087 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:17.575105 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.575985 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.575989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:17.577142 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:17.577152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.577745 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.577752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:17.577973 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:17.578640 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:17.578665 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:17.580302 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:17.582408 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:17.582417 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:17.582440 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:17.582848 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:17.583199 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:17.583202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:17.583203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:17.583350 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:17.583778 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:17.583786 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:17.584069 (+ 283us) client_negotiation.cc:769] Sending connection context
0925 22:44:17.584273 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 22:44:17.584639 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":124,"threads_started":1}
I20260925 22:44:17.586822 18729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:17.577621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52450 (local address 127.2.74.254:34049)
0925 22:44:17.578263 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:17.578267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:17.578277 (+ 10us) server_negotiation.cc:407] Connection header received
0925 22:44:17.578306 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:17.578308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:17.578342 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:17.578446 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:17.579258 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.579786 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.580906 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.581319 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.581427 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:17.582981 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:17.582986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:17.582989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:17.583031 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:17.584580 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:17.585074 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:17.585079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:17.585080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:17.585165 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:17.585460 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:17.585465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:17.585467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:17.585647 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:17.585889 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:17.586583 (+ 694us) server_negotiation.cc:299] Negotiation successful
0925 22:44:17.586699 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"spinlock_wait_cycles":3456,"thread_start_us":102,"threads_started":1}
I20260925 22:44:17.586925 18726 heartbeater.cc:347] Connected to a master server at 127.2.74.254:34049
I20260925 22:44:17.587224 18726 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:17.587836 18726 heartbeater.cc:511] Master 127.2.74.254:34049 requested a full tablet report, sending...
I20260925 22:44:17.589064 18268 ts_manager.cc:194] Registered new tserver with Master: c0a24c924c484ac2be5dbec2b5bb45e1 (127.2.74.195:43753)
I20260925 22:44:17.589591 18268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.195@KRBTEST.COM'} at 127.2.74.195:46779
I20260925 22:44:17.601059 2347 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:44:17.606685 18729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:17.601773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52462 (local address 127.2.74.254:34049)
0925 22:44:17.601953 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:17.601957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:17.602099 (+ 142us) server_negotiation.cc:407] Connection header received
0925 22:44:17.602243 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:17.602245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:17.602369 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:17.602467 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:44:17.602852 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.603657 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.604756 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.605057 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.605170 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:17.605659 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:44:17.605879 (+ 220us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 22:44:17.606519 (+ 640us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52462: token expired
Metrics: {"server-negotiator.queue_time_us":82}
W20260925 22:44:17.606801 18729 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52462: token expired
W20260925 22:44:17.607079 18196 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.74.254:34049)
I20260925 22:44:17.615799 18729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:17.607613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52468 (local address 127.2.74.254:34049)
0925 22:44:17.607759 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:17.607765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:17.607782 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:44:17.607910 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:17.607913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:17.607967 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:17.608104 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:17.608793 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.609412 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.610291 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.610521 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.610640 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:17.612264 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:17.612267 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:17.612270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:17.612299 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:17.613847 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:17.614362 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:17.614368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:17.614375 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:17.614457 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:17.614768 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:17.614773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:17.614774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:17.615063 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:17.615204 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:17.615471 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 22:44:17.615619 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:44:17.619529 18268 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:44:17.622925 18268 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:44:17.632341 18381 tablet_service.cc:1467] Processing CreateTablet for tablet ce32e77aacbe4a18925cc6a49ab7997b (DEFAULT_TABLE table=test-table [id=be8b901e15494661bbff46a1e49505ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:17.633082 18381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce32e77aacbe4a18925cc6a49ab7997b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:17.636582 18348 tablet_bootstrap.cc:492] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395: Bootstrap starting.
I20260925 22:44:17.637001 18463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:17.631027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:39625 (local address 127.0.0.1:60550)
0925 22:44:17.631167 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:17.631186 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:17.631275 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:17.632467 (+ 1192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:17.632470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:17.632491 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:17.632610 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:17.632616 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.634117 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.634121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:17.635364 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:17.635375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.636463 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.636468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:17.636590 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:17.636668 (+ 78us) client_negotiation.cc:769] Sending connection context
0925 22:44:17.636752 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 22:44:17.636854 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 22:44:17.637622 18736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:17.631333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60550 (local address 127.2.74.194:39625)
0925 22:44:17.631856 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:17.631872 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:17.631901 (+ 29us) server_negotiation.cc:407] Connection header received
0925 22:44:17.631987 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:17.631994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:17.632288 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:17.632470 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:17.632857 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.633977 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.635507 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.636301 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.636441 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:17.636623 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:17.637332 (+ 709us) server_negotiation.cc:299] Negotiation successful
0925 22:44:17.637456 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":131,"threads_started":1}
I20260925 22:44:17.637728 18348 tablet_bootstrap.cc:654] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:17.638463 18735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:17.631275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:43753 (local address 127.0.0.1:45514)
0925 22:44:17.631528 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:17.631542 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:17.631635 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:17.632834 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:17.632838 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:17.632857 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:17.632965 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:17.632971 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.634630 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.634634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:17.635754 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:17.635765 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.637969 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.637974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:17.638107 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:17.638129 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:44:17.638233 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 22:44:17.638321 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":120,"threads_started":1}
I20260925 22:44:17.638831 18737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:17.631484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45514 (local address 127.2.74.195:43753)
0925 22:44:17.632195 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:17.632207 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:17.632231 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:44:17.632307 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:17.632313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:17.632503 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:17.632648 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:17.633103 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.634394 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.635922 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.636763 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.636882 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:17.637040 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:17.638293 (+ 1253us) server_negotiation.cc:299] Negotiation successful
0925 22:44:17.638424 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":126,"threads_started":1}
I20260925 22:44:17.639747 18348 tablet_bootstrap.cc:492] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395: No bootstrap required, opened a new log
I20260925 22:44:17.639850 18348 ts_tablet_manager.cc:1397] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260925 22:44:17.640484 18348 raft_consensus.cc:348] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c767841753fa44429e8c8185fa5114fd" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39625 } } peers { permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 } } peers { permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 } }
I20260925 22:44:17.640763 18348 raft_consensus.cc:374] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:17.640856 18348 raft_consensus.cc:729] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bed4a0e54674a99ba4661ddb1ce6395, State: Initialized, Role: FOLLOWER
I20260925 22:44:17.641115 18348 consensus_queue.cc:260] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c767841753fa44429e8c8185fa5114fd" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39625 } } peers { permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 } } peers { permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 } }
I20260925 22:44:17.641382 18528 tablet_service.cc:1467] Processing CreateTablet for tablet ce32e77aacbe4a18925cc6a49ab7997b (DEFAULT_TABLE table=test-table [id=be8b901e15494661bbff46a1e49505ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:17.642431 18348 ts_tablet_manager.cc:1428] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260925 22:44:17.641956 18660 tablet_service.cc:1467] Processing CreateTablet for tablet ce32e77aacbe4a18925cc6a49ab7997b (DEFAULT_TABLE table=test-table [id=be8b901e15494661bbff46a1e49505ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:17.642736 18528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce32e77aacbe4a18925cc6a49ab7997b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:17.643213 18660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce32e77aacbe4a18925cc6a49ab7997b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:17.651161 18739 tablet_bootstrap.cc:492] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd: Bootstrap starting.
I20260925 22:44:17.651897 18740 tablet_bootstrap.cc:492] T ce32e77aacbe4a18925cc6a49ab7997b P c0a24c924c484ac2be5dbec2b5bb45e1: Bootstrap starting.
I20260925 22:44:17.653946 18739 tablet_bootstrap.cc:654] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:17.654446 18740 tablet_bootstrap.cc:654] T ce32e77aacbe4a18925cc6a49ab7997b P c0a24c924c484ac2be5dbec2b5bb45e1: Neither blocks nor log segments found. Creating new log.
W20260925 22:44:17.654485 18457 tablet.cc:2369] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:44:17.654876 18739 log.cc:824] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:17.655342 18740 log.cc:824] T ce32e77aacbe4a18925cc6a49ab7997b P c0a24c924c484ac2be5dbec2b5bb45e1: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:17.657316 18739 tablet_bootstrap.cc:492] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd: No bootstrap required, opened a new log
I20260925 22:44:17.657400 18740 tablet_bootstrap.cc:492] T ce32e77aacbe4a18925cc6a49ab7997b P c0a24c924c484ac2be5dbec2b5bb45e1: No bootstrap required, opened a new log
I20260925 22:44:17.657586 18739 ts_tablet_manager.cc:1397] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260925 22:44:17.657614 18740 ts_tablet_manager.cc:1397] T ce32e77aacbe4a18925cc6a49ab7997b P c0a24c924c484ac2be5dbec2b5bb45e1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:44:17.660825 18740 raft_consensus.cc:348] T ce32e77aacbe4a18925cc6a49ab7997b P c0a24c924c484ac2be5dbec2b5bb45e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c767841753fa44429e8c8185fa5114fd" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39625 } } peers { permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 } } peers { permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 } }
I20260925 22:44:17.660825 18739 raft_consensus.cc:348] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c767841753fa44429e8c8185fa5114fd" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39625 } } peers { permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 } } peers { permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 } }
I20260925 22:44:17.661128 18740 raft_consensus.cc:374] T ce32e77aacbe4a18925cc6a49ab7997b P c0a24c924c484ac2be5dbec2b5bb45e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:17.661157 18739 raft_consensus.cc:374] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:17.661170 18740 raft_consensus.cc:729] T ce32e77aacbe4a18925cc6a49ab7997b P c0a24c924c484ac2be5dbec2b5bb45e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0a24c924c484ac2be5dbec2b5bb45e1, State: Initialized, Role: FOLLOWER
I20260925 22:44:17.661237 18739 raft_consensus.cc:729] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c767841753fa44429e8c8185fa5114fd, State: Initialized, Role: FOLLOWER
I20260925 22:44:17.661732 18739 consensus_queue.cc:260] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c767841753fa44429e8c8185fa5114fd" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39625 } } peers { permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 } } peers { permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 } }
I20260925 22:44:17.661734 18740 consensus_queue.cc:260] T ce32e77aacbe4a18925cc6a49ab7997b P c0a24c924c484ac2be5dbec2b5bb45e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c767841753fa44429e8c8185fa5114fd" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39625 } } peers { permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 } } peers { permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 } }
I20260925 22:44:17.662540 18726 heartbeater.cc:503] Master 127.2.74.254:34049 was elected leader, sending a full tablet report...
I20260925 22:44:17.662940 18739 ts_tablet_manager.cc:1428] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260925 22:44:17.662920 18740 ts_tablet_manager.cc:1428] T ce32e77aacbe4a18925cc6a49ab7997b P c0a24c924c484ac2be5dbec2b5bb45e1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 22:44:17.663932 18594 heartbeater.cc:503] Master 127.2.74.254:34049 was elected leader, sending a full tablet report...
I20260925 22:44:17.665097 18743 raft_consensus.cc:482] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:44:17.665290 18743 raft_consensus.cc:504] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c767841753fa44429e8c8185fa5114fd" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39625 } } peers { permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 } } peers { permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 } }
I20260925 22:44:17.666684 18743 leader_election.cc:288] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0a24c924c484ac2be5dbec2b5bb45e1 (127.2.74.195:43753), 4bed4a0e54674a99ba4661ddb1ce6395 (127.2.74.193:37037)
I20260925 22:44:17.670799 18737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:17.667002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:43083 (local address 127.2.74.195:43753)
0925 22:44:17.667112 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:17.667119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:17.667173 (+ 54us) server_negotiation.cc:407] Connection header received
0925 22:44:17.667340 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:17.667345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:17.667393 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:17.667465 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:17.667824 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.668630 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.669746 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.670179 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.670321 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:17.670378 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:17.670576 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 22:44:17.670655 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 22:44:17.670799 18596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:17.666858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:43753 (local address 127.2.74.194:43083)
0925 22:44:17.667110 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:17.667128 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:17.667225 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:17.667591 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:17.667593 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:17.667609 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:17.667705 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:17.667710 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.668765 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.668768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:17.669593 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:17.669604 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.670317 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.670321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:17.670444 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:17.670463 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 22:44:17.670560 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 22:44:17.670648 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260925 22:44:17.670938 18464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:17.667442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:44983 (local address 127.2.74.193:37037)
0925 22:44:17.667605 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:17.667611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:17.667628 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:44:17.667688 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:17.667690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:17.667741 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:17.667874 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:17.668024 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.668822 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.669875 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.670360 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.670505 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:17.670685 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:17.670720 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 22:44:17.670803 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 22:44:17.671136 18745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:17.667145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:37037 (local address 127.2.74.194:44983)
0925 22:44:17.667449 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:17.667461 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:17.667574 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:17.667874 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:17.667875 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:17.667887 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:17.667955 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:17.667959 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.668939 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.668943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:17.669735 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:17.669744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.670496 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.670500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:17.670599 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:17.670621 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:44:17.670684 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 22:44:17.670762 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":120,"threads_started":1}
I20260925 22:44:17.671756 18680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce32e77aacbe4a18925cc6a49ab7997b" candidate_uuid: "c767841753fa44429e8c8185fa5114fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" is_pre_election: true
I20260925 22:44:17.671746 18410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce32e77aacbe4a18925cc6a49ab7997b" candidate_uuid: "c767841753fa44429e8c8185fa5114fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" is_pre_election: true
I20260925 22:44:17.672078 18680 raft_consensus.cc:2454] T ce32e77aacbe4a18925cc6a49ab7997b P c0a24c924c484ac2be5dbec2b5bb45e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c767841753fa44429e8c8185fa5114fd in term 0.
I20260925 22:44:17.672081 18410 raft_consensus.cc:2454] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c767841753fa44429e8c8185fa5114fd in term 0.
I20260925 22:44:17.672631 18483 leader_election.cc:302] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bed4a0e54674a99ba4661ddb1ce6395, c767841753fa44429e8c8185fa5114fd; no voters:
I20260925 22:44:17.672952 18743 raft_consensus.cc:2781] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:44:17.673058 18743 raft_consensus.cc:482] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:44:17.673116 18743 raft_consensus.cc:3037] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:17.674276 18743 raft_consensus.cc:504] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c767841753fa44429e8c8185fa5114fd" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39625 } } peers { permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 } } peers { permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 } }
I20260925 22:44:17.674651 18743 leader_election.cc:288] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd [CANDIDATE]: Term 1 election: Requested vote from peers c0a24c924c484ac2be5dbec2b5bb45e1 (127.2.74.195:43753), 4bed4a0e54674a99ba4661ddb1ce6395 (127.2.74.193:37037)
I20260925 22:44:17.675112 18410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce32e77aacbe4a18925cc6a49ab7997b" candidate_uuid: "c767841753fa44429e8c8185fa5114fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bed4a0e54674a99ba4661ddb1ce6395"
I20260925 22:44:17.675204 18680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce32e77aacbe4a18925cc6a49ab7997b" candidate_uuid: "c767841753fa44429e8c8185fa5114fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1"
I20260925 22:44:17.675299 18410 raft_consensus.cc:3037] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:17.675382 18680 raft_consensus.cc:3037] T ce32e77aacbe4a18925cc6a49ab7997b P c0a24c924c484ac2be5dbec2b5bb45e1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:17.676625 18410 raft_consensus.cc:2454] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c767841753fa44429e8c8185fa5114fd in term 1.
I20260925 22:44:17.676735 18680 raft_consensus.cc:2454] T ce32e77aacbe4a18925cc6a49ab7997b P c0a24c924c484ac2be5dbec2b5bb45e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c767841753fa44429e8c8185fa5114fd in term 1.
I20260925 22:44:17.676962 18483 leader_election.cc:302] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bed4a0e54674a99ba4661ddb1ce6395, c767841753fa44429e8c8185fa5114fd; no voters:
I20260925 22:44:17.677150 18743 raft_consensus.cc:2781] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:17.677386 18743 raft_consensus.cc:686] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd [term 1 LEADER]: Becoming Leader. State: Replica: c767841753fa44429e8c8185fa5114fd, State: Running, Role: LEADER
I20260925 22:44:17.677755 18743 consensus_queue.cc:237] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c767841753fa44429e8c8185fa5114fd" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39625 } } peers { permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 } } peers { permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 } }
I20260925 22:44:17.681552 18268 catalog_manager.cc:5495] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c767841753fa44429e8c8185fa5114fd (127.2.74.194). New cstate: current_term: 1 leader_uuid: "c767841753fa44429e8c8185fa5114fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c767841753fa44429e8c8185fa5114fd" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 } health_report { overall_health: UNKNOWN } } }
I20260925 22:44:17.708199 18736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:17.702599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60566 (local address 127.2.74.194:39625)
0925 22:44:17.702761 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:17.702766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:17.702781 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:44:17.703012 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:17.703015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:17.703087 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:17.703366 (+ 279us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:44:17.703786 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.704539 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.705602 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:17.705852 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:17.705989 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:17.706745 (+ 756us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:44:17.707537 (+ 792us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:44:17.707712 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:17.707834 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 22:44:17.707933 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 22:44:17.716794 18680 raft_consensus.cc:1261] T ce32e77aacbe4a18925cc6a49ab7997b P c0a24c924c484ac2be5dbec2b5bb45e1 [term 1 FOLLOWER]: Refusing update from remote peer c767841753fa44429e8c8185fa5114fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:17.716790 18410 raft_consensus.cc:1261] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [term 1 FOLLOWER]: Refusing update from remote peer c767841753fa44429e8c8185fa5114fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:17.717758 18746 consensus_queue.cc:1035] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:17.718101 18743 consensus_queue.cc:1035] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:17.724898 18748 mvcc.cc:205] Tried to move back new op lower bound from 7333381151596998656 to 7333381151451918336. Current Snapshot: MvccSnapshot[applied={T|T < 7333381151596998656}]
I20260925 22:44:17.736083 18268 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:17.736265 18268 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:17.738994 18268 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9: Sending DeleteTablet for 3 replicas of tablet ce32e77aacbe4a18925cc6a49ab7997b
I20260925 22:44:17.739847 18528 tablet_service.cc:1514] Processing DeleteTablet for tablet ce32e77aacbe4a18925cc6a49ab7997b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:17 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:60550
I20260925 22:44:17.739943 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 18306
I20260925 22:44:17.740145 18660 tablet_service.cc:1514] Processing DeleteTablet for tablet ce32e77aacbe4a18925cc6a49ab7997b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:17 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:45514
I20260925 22:44:17.741751 18759 tablet_replica.cc:330] T ce32e77aacbe4a18925cc6a49ab7997b P c0a24c924c484ac2be5dbec2b5bb45e1: stopping tablet replica
I20260925 22:44:17.741982 18758 tablet_replica.cc:330] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd: stopping tablet replica
I20260925 22:44:17.742156 18759 raft_consensus.cc:2229] T ce32e77aacbe4a18925cc6a49ab7997b P c0a24c924c484ac2be5dbec2b5bb45e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:17.742226 18758 raft_consensus.cc:2229] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:44:17.742517 18759 raft_consensus.cc:2258] T ce32e77aacbe4a18925cc6a49ab7997b P c0a24c924c484ac2be5dbec2b5bb45e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:17.742604 18758 raft_consensus.cc:2258] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:17.744284 18759 ts_tablet_manager.cc:1905] T ce32e77aacbe4a18925cc6a49ab7997b P c0a24c924c484ac2be5dbec2b5bb45e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:17.744284 18758 ts_tablet_manager.cc:1905] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:17.747406 18759 ts_tablet_manager.cc:1918] T ce32e77aacbe4a18925cc6a49ab7997b P c0a24c924c484ac2be5dbec2b5bb45e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:44:17.747417 18758 ts_tablet_manager.cc:1918] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:44:17.747550 18759 log.cc:1196] T ce32e77aacbe4a18925cc6a49ab7997b P c0a24c924c484ac2be5dbec2b5bb45e1: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/wal/wals/ce32e77aacbe4a18925cc6a49ab7997b
I20260925 22:44:17.747920 18759 ts_tablet_manager.cc:1939] T ce32e77aacbe4a18925cc6a49ab7997b P c0a24c924c484ac2be5dbec2b5bb45e1: Deleting consensus metadata
W20260925 22:44:17.747991 18254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.74.193:37037 (error 108)
I20260925 22:44:17.748176 18758 log.cc:1196] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/wal/wals/ce32e77aacbe4a18925cc6a49ab7997b
I20260925 22:44:17.749277 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 18460
I20260925 22:44:17.749943 18254 catalog_manager.cc:4841] TS c0a24c924c484ac2be5dbec2b5bb45e1 (127.2.74.195:43753): tablet ce32e77aacbe4a18925cc6a49ab7997b (table test-table [id=be8b901e15494661bbff46a1e49505ae]) successfully deleted
I20260925 22:44:17.756623 18735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:17.756175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:37037 (local address 127.0.0.1:35584)
0925 22:44:17.756384 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:17.756491 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.193:37037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 22:44:17.756950 18254 catalog_manager.cc:4584] TS 4bed4a0e54674a99ba4661ddb1ce6395 (127.2.74.193:37037): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ce32e77aacbe4a18925cc6a49ab7997b: Network error: Client connection negotiation failed: client connection to 127.2.74.193:37037: connect: Connection refused (error 111)
I20260925 22:44:17.760861 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 18597
I20260925 22:44:17.762390 18735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:17.762070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:39625 (local address 127.0.0.1:60574)
0925 22:44:17.762232 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:17.762300 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.194:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 22:44:17.768527 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 18235
I20260925 22:44:37.779065 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:34049
--webserver_interface=127.2.74.254
--webserver_port=43585
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:43275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:34049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:37.899048 18761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:37.899397 18761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:37.899479 18761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:37.903748 18761 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 22:44:37.903841 18761 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 22:44:37.903867 18761 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 22:44:37.903898 18761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:44:37.903944 18761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:37.903986 18761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:37.904038 18761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:44:37.904079 18761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:44:37.909565 18761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:43275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:34049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:34049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=43585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:44:37.910751 18761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:37.911882 18761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:44:37.920846 18761 server_base.cc:1034] running on GCE node
W20260925 22:44:37.920874 18767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:37.920877 18766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:37.920877 18769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:37.921880 18761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:37.923143 18761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:37.924356 18761 hybrid_clock.cc:648] HybridClock initialized: now 1790376277924340 us; error 32 us; skew 500 ppm
Sep 25 22:44:37 dist-test-slave-csnl krb5kdc[17688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376277, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:37.933293 18761 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:44:37.934432 18761 webserver.cc:460] Webserver started at http://127.2.74.254:43585/ using document root <none> and password file <none>
I20260925 22:44:37.935163 18761 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:37.935277 18761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:37.939695 18761 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:37.942173 18776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:37.943751 18761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:37.943930 18761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "0e78555668e843efbb9de814ef9dcbe9"
format_stamp: "Formatted at 2026-09-25 22:43:55 on dist-test-slave-csnl"
server_key: "e63f0d422b6e8cce8d9c5b6516d9fddd"
server_key_iv: "663e048941c2f3de57ed20ebb28df4a4"
server_key_version: "encryptionkey@0"
I20260925 22:44:37.944585 18761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:37.960325 18761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:37.964210 18761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:37.964457 18761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:37.972312 18761 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:34049
I20260925 22:44:37.972618 18828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:34049 every 8 connection(s)
I20260925 22:44:37.973585 18761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:44:37.976166 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 18761
I20260925 22:44:37.976697 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:37037
--local_ip_for_outbound_sockets=127.2.74.193
--tserver_master_addrs=127.2.74.254:34049
--webserver_port=44489
--webserver_interface=127.2.74.193
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:43275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:44:37.978670 18829 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:44:37.980691 18829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9: Bootstrap starting.
I20260925 22:44:37.998560 18829 log.cc:824] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:38.010392 18829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:44:38.010857 18829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9: Bootstrap complete.
I20260925 22:44:38.015128 18829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e78555668e843efbb9de814ef9dcbe9" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34049 } }
I20260925 22:44:38.015618 18829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e78555668e843efbb9de814ef9dcbe9, State: Initialized, Role: FOLLOWER
I20260925 22:44:38.016201 18829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e78555668e843efbb9de814ef9dcbe9" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34049 } }
I20260925 22:44:38.016369 18829 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:44:38.016463 18829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:44:38.016589 18829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [term 2 FOLLOWER]: Advancing to term 3
I20260925 22:44:38.018747 18829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e78555668e843efbb9de814ef9dcbe9" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34049 } }
I20260925 22:44:38.019187 18829 leader_election.cc:302] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e78555668e843efbb9de814ef9dcbe9; no voters:
I20260925 22:44:38.019587 18829 leader_election.cc:288] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 22:44:38.019693 18833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 22:44:38.019910 18833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [term 3 LEADER]: Becoming Leader. State: Replica: 0e78555668e843efbb9de814ef9dcbe9, State: Running, Role: LEADER
I20260925 22:44:38.020260 18833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e78555668e843efbb9de814ef9dcbe9" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34049 } }
I20260925 22:44:38.021013 18829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:44:38.021831 18834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0e78555668e843efbb9de814ef9dcbe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e78555668e843efbb9de814ef9dcbe9" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34049 } } }
I20260925 22:44:38.022004 18834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:38.022696 18835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e78555668e843efbb9de814ef9dcbe9. Latest consensus state: current_term: 3 leader_uuid: "0e78555668e843efbb9de814ef9dcbe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e78555668e843efbb9de814ef9dcbe9" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34049 } } }
I20260925 22:44:38.022862 18835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9 [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:38.023123 18842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:44:38.029012 18842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:44:38.029770 18842 catalog_manager.cc:1259] Loaded cluster ID: b95bcf3b1ddc4e09a3b9423c1f0db741
I20260925 22:44:38.029865 18842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:44:38.032631 18842 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:44:38.033587 18842 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9: Loaded TSK: 0
I20260925 22:44:38.033699 18842 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0e78555668e843efbb9de814ef9dcbe9: Loaded TSK: 1
F20260925 22:44:38.034641 18842 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9417ca59ad google::LogMessage::Fail() at ??:0
@ 0x7f9417ca888a google::LogMessage::SendToLog() at ??:0
@ 0x7f9417ca54de google::LogMessage::Flush() at ??:0
@ 0x7f9417ca6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f941a96918d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f941a961f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f942260a7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f94225e499c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f94225e5178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f9422626df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f942264582b std::function<>::operator()() at ??:0
@ 0x7f94225e4b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f94225e5f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f94225e2579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f9422626640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f9419ef73d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9419ef7c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f9419ef93f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f9419ee8512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f941b52c6db start_thread at ??:0
@ 0x7f94167a361f clone at ??:0
W20260925 22:44:38.138527 18831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:38.138813 18831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:38.138859 18831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:38.143291 18831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:38.143355 18831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:38.143472 18831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:44:38.149628 18831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:43275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:37037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=44489
--tserver_master_addrs=127.2.74.254:34049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:44:38.150808 18831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:38.151929 18831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:38.162200 18877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:38.162209 18876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:38.162596 18879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:38.162761 18831 server_base.cc:1034] running on GCE node
I20260925 22:44:38.163341 18831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:38.164268 18831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:38.165494 18831 hybrid_clock.cc:648] HybridClock initialized: now 1790376278165467 us; error 37 us; skew 500 ppm
Sep 25 22:44:38 dist-test-slave-csnl krb5kdc[17688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376278, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:38.169735 18831 init.cc:377] Logged in from keytab as kudu/127.2.74.193@KRBTEST.COM (short username kudu)
I20260925 22:44:38.171103 18831 webserver.cc:460] Webserver started at http://127.2.74.193:44489/ using document root <none> and password file <none>
I20260925 22:44:38.171983 18831 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:38.172101 18831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:38.178242 18831 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 22:44:38.181422 18886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:38.183243 18831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:38.183406 18831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "4bed4a0e54674a99ba4661ddb1ce6395"
format_stamp: "Formatted at 2026-09-25 22:43:56 on dist-test-slave-csnl"
server_key: "62831131f6d0b5de48446fa4ad439ff8"
server_key_iv: "f5dbf9d44f655741b1882dbcd3840308"
server_key_version: "encryptionkey@0"
I20260925 22:44:38.184258 18831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:38.211242 18831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:38.215476 18831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:38.215718 18831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:38.216524 18831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:38.218039 18893 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 22:44:38.224100 18831 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 22:44:38.224221 18831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260925 22:44:38.224308 18831 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 22:44:38.226858 18831 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 22:44:38.226938 18831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260925 22:44:38.227324 18893 tablet_bootstrap.cc:492] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395: Bootstrap starting.
I20260925 22:44:38.238619 18831 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:37037
I20260925 22:44:38.238941 19000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:37037 every 8 connection(s)
I20260925 22:44:38.240090 18831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:44:38.245285 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 18831
I20260925 22:44:38.245855 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:39625
--local_ip_for_outbound_sockets=127.2.74.194
--tserver_master_addrs=127.2.74.254:34049
--webserver_port=33473
--webserver_interface=127.2.74.194
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:43275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:44:38.248751 18893 log.cc:824] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:38.259745 18893 tablet_bootstrap.cc:492] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:44:38.260241 18893 tablet_bootstrap.cc:492] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395: Bootstrap complete.
I20260925 22:44:38.261734 18893 ts_tablet_manager.cc:1397] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395: Time spent bootstrapping tablet: real 0.035s user 0.021s sys 0.011s
I20260925 22:44:38.265206 18893 raft_consensus.cc:348] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c767841753fa44429e8c8185fa5114fd" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39625 } } peers { permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 } } peers { permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 } }
I20260925 22:44:38.265797 18893 raft_consensus.cc:729] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bed4a0e54674a99ba4661ddb1ce6395, State: Initialized, Role: FOLLOWER
I20260925 22:44:38.266399 18893 consensus_queue.cc:260] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c767841753fa44429e8c8185fa5114fd" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39625 } } peers { permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 } } peers { permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 } }
I20260925 22:44:38.267562 18893 ts_tablet_manager.cc:1428] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
W20260925 22:44:38.292388 19003 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:44:38.242678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.193:59571)
0925 22:44:38.243387 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:38.243462 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:38.244657 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:38.291593 (+ 46936us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":180,"threads_started":1}
W20260925 22:44:38.292910 19001 heartbeater.cc:644] Failed to heartbeat to 127.2.74.254:34049 (0 consecutive failures): Network error: Failed to ping master at 127.2.74.254:34049: Client connection negotiation failed: client connection to 127.2.74.254:34049: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 22:44:38.293681 19003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:38.293407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.193:44863)
0925 22:44:38.293540 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:38.293620 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 22:44:38.294332 19003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:38.294121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.193:53193)
0925 22:44:38.294229 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:38.294285 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 22:44:38.294492 19001 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:44:38.372144 19004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:38.372494 19004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:38.372592 19004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:38.376932 19004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:38.377071 19004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:38.377172 19004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:44:38.382925 19004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:43275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:39625
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=33473
--tserver_master_addrs=127.2.74.254:34049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:44:38.384217 19004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:38.385339 19004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:38.396399 19011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:38.396443 19014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:38.396607 19004 server_base.cc:1034] running on GCE node
W20260925 22:44:38.396462 19012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:38.397195 19004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:38.397919 19004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:38.399129 19004 hybrid_clock.cc:648] HybridClock initialized: now 1790376278399082 us; error 56 us; skew 500 ppm
Sep 25 22:44:38 dist-test-slave-csnl krb5kdc[17688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376278, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:38.402246 19004 init.cc:377] Logged in from keytab as kudu/127.2.74.194@KRBTEST.COM (short username kudu)
I20260925 22:44:38.403369 19004 webserver.cc:460] Webserver started at http://127.2.74.194:33473/ using document root <none> and password file <none>
I20260925 22:44:38.403962 19004 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:38.404030 19004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:38.408445 19004 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:44:38.410781 19021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:38.412298 19004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:38.412487 19004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "c767841753fa44429e8c8185fa5114fd"
format_stamp: "Formatted at 2026-09-25 22:43:56 on dist-test-slave-csnl"
server_key: "bf3482e6a04f9d506d0dfede7a345283"
server_key_iv: "51fbcdda3594314272adb636f7511411"
server_key_version: "encryptionkey@0"
I20260925 22:44:38.413116 19004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:38.434222 19004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:38.437942 19004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:38.438143 19004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:38.438814 19004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:38.440013 19028 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260925 22:44:38.443969 19028 ts_tablet_manager.cc:1852] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260925 22:44:38.444425 19028 ts_tablet_manager.cc:1905] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:38.448611 19028 ts_tablet_manager.cc:1918] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:44:38.448766 19028 log.cc:1196] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/wal/wals/ce32e77aacbe4a18925cc6a49ab7997b
I20260925 22:44:38.449139 19028 ts_tablet_manager.cc:1939] T ce32e77aacbe4a18925cc6a49ab7997b P c767841753fa44429e8c8185fa5114fd: Deleting consensus metadata
I20260925 22:44:38.449489 19004 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260925 22:44:38.449558 19004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.000s
I20260925 22:44:38.449663 19004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:38.449710 19004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:38.459844 19004 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:39625
I20260925 22:44:38.459878 19135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:39625 every 8 connection(s)
I20260925 22:44:38.460872 19004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:44:38.465113 19138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:38.463369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.194:53429)
0925 22:44:38.464025 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:38.464151 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":172,"threads_started":1}
I20260925 22:44:38.465210 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 19004
W20260925 22:44:38.465634 19136 heartbeater.cc:644] Failed to heartbeat to 127.2.74.254:34049 (0 consecutive failures): Network error: Failed to ping master at 127.2.74.254:34049: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
I20260925 22:44:38.465694 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:43753
--local_ip_for_outbound_sockets=127.2.74.195
--tserver_master_addrs=127.2.74.254:34049
--webserver_port=36811
--webserver_interface=127.2.74.195
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:43275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:44:38.466385 19138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:38.466056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.194:38537)
0925 22:44:38.466218 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:38.466291 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 22:44:38.467130 19138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:38.466834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.194:58641)
0925 22:44:38.466986 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:38.467050 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 22:44:38.467303 19136 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:44:38.582715 19139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:38.583086 19139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:38.583137 19139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:38.587378 19139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:38.587452 19139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:38.587517 19139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:44:38.593252 19139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:43275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:43753
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=36811
--tserver_master_addrs=127.2.74.254:34049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:44:38.594348 19139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:38.595489 19139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:38.605883 19144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:38.606102 19139 server_base.cc:1034] running on GCE node
W20260925 22:44:38.605908 19145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:38.605896 19147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:38.606714 19139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:38.607560 19139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:38.608762 19139 hybrid_clock.cc:648] HybridClock initialized: now 1790376278608726 us; error 52 us; skew 500 ppm
Sep 25 22:44:38 dist-test-slave-csnl krb5kdc[17688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376278, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:38.611908 19139 init.cc:377] Logged in from keytab as kudu/127.2.74.195@KRBTEST.COM (short username kudu)
I20260925 22:44:38.613046 19139 webserver.cc:460] Webserver started at http://127.2.74.195:36811/ using document root <none> and password file <none>
I20260925 22:44:38.613665 19139 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:38.613741 19139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:38.618228 19139 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:44:38.620761 19154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:38.622229 19139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:44:38.622416 19139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "c0a24c924c484ac2be5dbec2b5bb45e1"
format_stamp: "Formatted at 2026-09-25 22:43:56 on dist-test-slave-csnl"
server_key: "df8405a897aaa0f23d64580c402fbd5f"
server_key_iv: "ec828e616be0374ba2f487fba9699aab"
server_key_version: "encryptionkey@0"
I20260925 22:44:38.623101 19139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:38.644434 19139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:38.647933 19139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:38.648102 19139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:38.648694 19139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:38.649510 19139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:38.649559 19139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:38.649600 19139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:38.649614 19139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:38.659382 19139 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:43753
I20260925 22:44:38.659400 19267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:43753 every 8 connection(s)
I20260925 22:44:38.660547 19139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:44:38.663348 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 19139
I20260925 22:44:38.664871 19270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:38.662858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.195:44583)
0925 22:44:38.663806 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:38.664051 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":122,"threads_started":1}
W20260925 22:44:38.665454 19268 heartbeater.cc:644] Failed to heartbeat to 127.2.74.254:34049 (0 consecutive failures): Network error: Failed to ping master at 127.2.74.254:34049: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
I20260925 22:44:38.666006 19270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:38.665793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.195:50285)
0925 22:44:38.665904 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:38.665951 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 22:44:38.666589 19270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:38.666413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.195:60721)
0925 22:44:38.666502 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:38.666541 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 22:44:38.666774 19268 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 22:44:39.295513 19272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:39.295089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.193:39745)
0925 22:44:39.295339 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:39.295396 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":73,"threads_started":1}
I20260925 22:44:39.468421 19273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:39.467961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.194:40135)
0925 22:44:39.468260 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:39.468323 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":75,"threads_started":1}
I20260925 22:44:39.667963 19274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:39.667505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.195:54307)
0925 22:44:39.667783 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:39.667850 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":85,"threads_started":1}
I20260925 22:44:39.915015 19275 raft_consensus.cc:482] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:44:39.915294 19275 raft_consensus.cc:504] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c767841753fa44429e8c8185fa5114fd" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39625 } } peers { permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 } } peers { permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 } }
I20260925 22:44:39.916510 19275 leader_election.cc:288] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c767841753fa44429e8c8185fa5114fd (127.2.74.194:39625), c0a24c924c484ac2be5dbec2b5bb45e1 (127.2.74.195:43753)
Sep 25 22:44:39 dist-test-slave-csnl krb5kdc[17688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376278, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for kudu/127.2.74.194@KRBTEST.COM
Sep 25 22:44:39 dist-test-slave-csnl krb5kdc[17688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376278, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for kudu/127.2.74.195@KRBTEST.COM
I20260925 22:44:39.931224 19278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:39.917153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:43753 (local address 127.2.74.193:35211)
0925 22:44:39.917397 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:39.917411 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:39.917547 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:39.919033 (+ 1486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:39.919038 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:39.919610 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:39.920100 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:39.920117 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:39.921413 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:39.921417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:39.922362 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:39.922372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:39.923357 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:39.923359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:39.923434 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:39.924874 (+ 1440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:39.924891 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:39.926458 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:39.928831 (+ 2373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:39.928834 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:39.928835 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:39.929142 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:39.929534 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:39.929537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:39.929539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:39.929644 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:39.930264 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:39.930266 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:39.930418 (+ 152us) client_negotiation.cc:769] Sending connection context
0925 22:44:39.930611 (+ 193us) client_negotiation.cc:240] Negotiation successful
0925 22:44:39.931012 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":165,"spinlock_wait_cycles":1920,"thread_start_us":54,"threads_started":1}
I20260925 22:44:39.931237 19277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:39.916905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:42781 (local address 127.2.74.194:39625)
0925 22:44:39.917376 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:39.917392 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:39.917440 (+ 48us) server_negotiation.cc:407] Connection header received
0925 22:44:39.918243 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:39.918249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:39.918622 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:39.918933 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:39.920291 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:39.921257 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:39.922494 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:39.923016 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:39.923183 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:39.926326 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:39.926333 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:39.926351 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:39.926389 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:39.928316 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:39.928988 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:39.928995 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:39.929001 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:39.929153 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:39.929534 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:39.929538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:39.929539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:39.930042 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:39.930148 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:39.930723 (+ 575us) server_negotiation.cc:299] Negotiation successful
0925 22:44:39.931018 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":136,"threads_started":1}
I20260925 22:44:39.931255 19279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:39.917334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:35211 (local address 127.2.74.195:43753)
0925 22:44:39.917638 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:39.917655 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:39.917684 (+ 29us) server_negotiation.cc:407] Connection header received
0925 22:44:39.918403 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:39.918410 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:39.918741 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:39.919040 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:39.920293 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:39.921259 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:39.922504 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:39.923026 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:39.923210 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:39.926657 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:39.926663 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:39.926674 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:39.926703 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:39.928706 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:39.929272 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:39.929278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:39.929283 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:39.929419 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:39.929734 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:39.929737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:39.929738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:39.930152 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:39.930278 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:39.930737 (+ 459us) server_negotiation.cc:299] Negotiation successful
0925 22:44:39.931056 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":103,"threads_started":1}
I20260925 22:44:39.931252 19276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:39.916757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:39625 (local address 127.2.74.193:42781)
0925 22:44:39.917039 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:39.917058 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:39.917179 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:39.919032 (+ 1853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:39.919039 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:39.919609 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:39.920100 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:39.920117 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:39.921417 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:39.921421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:39.922359 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:39.922369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:39.923237 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:39.923242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:39.923424 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:39.924179 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:39.924209 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:39.926028 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:39.928461 (+ 2433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:39.928470 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:39.928487 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:39.928865 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:39.929256 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:39.929260 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:39.929262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:39.929439 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:39.930165 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:39.930172 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:39.930420 (+ 248us) client_negotiation.cc:769] Sending connection context
0925 22:44:39.930612 (+ 192us) client_negotiation.cc:240] Negotiation successful
0925 22:44:39.931083 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":230,"spinlock_wait_cycles":3328,"thread_start_us":96,"threads_started":1}
W20260925 22:44:39.934327 18889 leader_election.cc:341] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c0a24c924c484ac2be5dbec2b5bb45e1 (127.2.74.195:43753): Not found: Tablet not found: ce32e77aacbe4a18925cc6a49ab7997b
W20260925 22:44:39.934875 18889 leader_election.cc:341] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c767841753fa44429e8c8185fa5114fd (127.2.74.194:39625): Not found: Tablet not found: ce32e77aacbe4a18925cc6a49ab7997b
I20260925 22:44:39.935020 18889 leader_election.cc:302] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bed4a0e54674a99ba4661ddb1ce6395; no voters: c0a24c924c484ac2be5dbec2b5bb45e1, c767841753fa44429e8c8185fa5114fd
I20260925 22:44:39.935343 19275 raft_consensus.cc:2726] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:44:40.296864 19276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:40.296417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.193:41143)
0925 22:44:40.296640 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:40.296743 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260925 22:44:40.469883 19280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:40.469380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.194:35691)
0925 22:44:40.469698 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:40.469782 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":94,"threads_started":1}
I20260925 22:44:40.669516 19281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:40.669008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.195:55325)
0925 22:44:40.669358 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:40.669418 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":129,"threads_started":1}
I20260925 22:44:41.298305 19282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:41.297750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.193:52269)
0925 22:44:41.298132 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:41.298200 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":76,"threads_started":1}
I20260925 22:44:41.471243 19283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:41.470756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.194:33831)
0925 22:44:41.471090 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:41.471148 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":129,"threads_started":1}
I20260925 22:44:41.670682 19284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:41.670271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.195:40185)
0925 22:44:41.670516 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:41.670572 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":73,"threads_started":1}
I20260925 22:44:42.093134 19285 raft_consensus.cc:482] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:44:42.093317 19285 raft_consensus.cc:504] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c767841753fa44429e8c8185fa5114fd" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39625 } } peers { permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 } } peers { permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 } }
I20260925 22:44:42.093762 19285 leader_election.cc:288] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c767841753fa44429e8c8185fa5114fd (127.2.74.194:39625), c0a24c924c484ac2be5dbec2b5bb45e1 (127.2.74.195:43753)
W20260925 22:44:42.094738 18889 leader_election.cc:341] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c767841753fa44429e8c8185fa5114fd (127.2.74.194:39625): Not found: Tablet not found: ce32e77aacbe4a18925cc6a49ab7997b
W20260925 22:44:42.094918 18889 leader_election.cc:341] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c0a24c924c484ac2be5dbec2b5bb45e1 (127.2.74.195:43753): Not found: Tablet not found: ce32e77aacbe4a18925cc6a49ab7997b
I20260925 22:44:42.095081 18889 leader_election.cc:302] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bed4a0e54674a99ba4661ddb1ce6395; no voters: c0a24c924c484ac2be5dbec2b5bb45e1, c767841753fa44429e8c8185fa5114fd
I20260925 22:44:42.095296 19285 raft_consensus.cc:2726] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:44:42.299678 19286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:42.299245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.193:45435)
0925 22:44:42.299523 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:42.299579 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":92,"threads_started":1}
I20260925 22:44:42.472645 19287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:42.472129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.194:33865)
0925 22:44:42.472460 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:42.472528 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":99,"threads_started":1}
I20260925 22:44:42.672088 19288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:42.671616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.195:55841)
0925 22:44:42.671920 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:42.671993 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":81,"threads_started":1}
I20260925 22:44:43.301173 19289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:43.300684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.193:44923)
0925 22:44:43.301006 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:43.301073 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":78,"threads_started":1}
I20260925 22:44:43.474025 19290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:43.473570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.194:38639)
0925 22:44:43.473859 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:43.473925 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":84,"threads_started":1}
I20260925 22:44:43.673547 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:43.673063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.195:58431)
0925 22:44:43.673377 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:43.673447 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":88,"threads_started":1}
I20260925 22:44:44.302518 19292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:44.302025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.193:53801)
0925 22:44:44.302328 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:44.302406 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":86,"threads_started":1}
I20260925 22:44:44.475530 19293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:44.475086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.194:50435)
0925 22:44:44.475358 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:44.475422 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260925 22:44:44.675117 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:44.674560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.195:47875)
0925 22:44:44.674866 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:44.674940 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":138,"threads_started":1}
I20260925 22:44:44.806864 19295 raft_consensus.cc:482] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:44:44.807083 19295 raft_consensus.cc:504] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c767841753fa44429e8c8185fa5114fd" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39625 } } peers { permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 } } peers { permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 } }
I20260925 22:44:44.807572 19295 leader_election.cc:288] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c767841753fa44429e8c8185fa5114fd (127.2.74.194:39625), c0a24c924c484ac2be5dbec2b5bb45e1 (127.2.74.195:43753)
W20260925 22:44:44.808409 18889 leader_election.cc:341] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c0a24c924c484ac2be5dbec2b5bb45e1 (127.2.74.195:43753): Not found: Tablet not found: ce32e77aacbe4a18925cc6a49ab7997b
W20260925 22:44:44.808547 18889 leader_election.cc:341] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c767841753fa44429e8c8185fa5114fd (127.2.74.194:39625): Not found: Tablet not found: ce32e77aacbe4a18925cc6a49ab7997b
I20260925 22:44:44.808600 18889 leader_election.cc:302] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bed4a0e54674a99ba4661ddb1ce6395; no voters: c0a24c924c484ac2be5dbec2b5bb45e1, c767841753fa44429e8c8185fa5114fd
I20260925 22:44:44.808758 19295 raft_consensus.cc:2726] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:44:45.304032 19296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:45.303508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.193:34651)
0925 22:44:45.303829 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:45.303916 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":91,"threads_started":1}
I20260925 22:44:45.477023 19297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:45.476428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.194:55847)
0925 22:44:45.476804 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:45.476875 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":95,"threads_started":1}
I20260925 22:44:45.676496 19298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:45.675926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.195:36481)
0925 22:44:45.676287 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:45.676354 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":99,"threads_started":1}
I20260925 22:44:46.305366 19299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:46.304942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.193:38499)
0925 22:44:46.305205 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:46.305269 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":78,"threads_started":1}
I20260925 22:44:46.478235 19300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:46.477815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.194:53169)
0925 22:44:46.478075 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:46.478133 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":79,"threads_started":1}
I20260925 22:44:46.677716 19301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:46.677285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.195:59257)
0925 22:44:46.677555 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:46.677619 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":74,"threads_started":1}
I20260925 22:44:47.306737 19302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:47.306236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.193:39909)
0925 22:44:47.306513 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:47.306578 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":81,"threads_started":1}
I20260925 22:44:47.479588 19303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:47.479138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.194:48439)
0925 22:44:47.479420 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:47.479480 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":77,"threads_started":1}
I20260925 22:44:47.679190 19304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:47.678659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.195:59377)
0925 22:44:47.679019 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:47.679090 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":85,"threads_started":1}
I20260925 22:44:48.308167 19305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:48.307732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.193:56691)
0925 22:44:48.308005 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:48.308069 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":78,"threads_started":1}
I20260925 22:44:48.480942 19306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:48.480492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.194:52961)
0925 22:44:48.480765 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:48.480829 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":81,"threads_started":1}
I20260925 22:44:48.680680 19307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:48.680241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.195:53375)
0925 22:44:48.680518 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:48.680583 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":78,"threads_started":1}
I20260925 22:44:48.733527 19308 raft_consensus.cc:482] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:44:48.733700 19308 raft_consensus.cc:504] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c767841753fa44429e8c8185fa5114fd" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39625 } } peers { permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 } } peers { permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 } }
I20260925 22:44:48.734148 19308 leader_election.cc:288] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c767841753fa44429e8c8185fa5114fd (127.2.74.194:39625), c0a24c924c484ac2be5dbec2b5bb45e1 (127.2.74.195:43753)
W20260925 22:44:48.735054 18889 leader_election.cc:341] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c767841753fa44429e8c8185fa5114fd (127.2.74.194:39625): Not found: Tablet not found: ce32e77aacbe4a18925cc6a49ab7997b
W20260925 22:44:48.735272 18889 leader_election.cc:341] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c0a24c924c484ac2be5dbec2b5bb45e1 (127.2.74.195:43753): Not found: Tablet not found: ce32e77aacbe4a18925cc6a49ab7997b
I20260925 22:44:48.735337 18889 leader_election.cc:302] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bed4a0e54674a99ba4661ddb1ce6395; no voters: c0a24c924c484ac2be5dbec2b5bb45e1, c767841753fa44429e8c8185fa5114fd
I20260925 22:44:48.735550 19308 raft_consensus.cc:2726] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:44:49.309547 19309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:49.309091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.193:45989)
0925 22:44:49.309379 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:49.309450 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":80,"threads_started":1}
I20260925 22:44:49.482463 19310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:49.481891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.194:38833)
0925 22:44:49.482270 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:49.482358 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":78,"threads_started":1}
I20260925 22:44:49.682075 19311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:49.681589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.195:59809)
0925 22:44:49.681882 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:49.681949 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":96,"threads_started":1}
I20260925 22:44:50.311098 19312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:50.310559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.193:55777)
0925 22:44:50.310881 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:50.310984 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":95,"threads_started":1}
I20260925 22:44:50.483907 19313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:50.483455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.194:55723)
0925 22:44:50.483745 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:50.483807 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":87,"threads_started":1}
I20260925 22:44:50.683492 19314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:50.683044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.195:50291)
0925 22:44:50.683324 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:50.683394 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":117,"threads_started":1}
I20260925 22:44:51.312647 19315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:51.311963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.193:48199)
0925 22:44:51.312456 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:51.312529 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":80,"threads_started":1}
I20260925 22:44:51.485421 19316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:51.484910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.194:58217)
0925 22:44:51.485254 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:51.485322 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":110,"threads_started":1}
I20260925 22:44:51.684753 19317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:51.684292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.195:48395)
0925 22:44:51.684584 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:51.684652 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":75,"threads_started":1}
I20260925 22:44:52.313897 19318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:52.313468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.193:53793)
0925 22:44:52.313733 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:52.313803 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":94,"threads_started":1}
I20260925 22:44:52.486868 19319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:52.486405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.194:44637)
0925 22:44:52.486697 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:52.486766 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":81,"threads_started":1}
I20260925 22:44:52.686195 19320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:52.685674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.195:49513)
0925 22:44:52.686031 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:52.686097 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":122,"threads_started":1}
I20260925 22:44:53.037384 19321 raft_consensus.cc:482] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:44:53.037540 19321 raft_consensus.cc:504] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c767841753fa44429e8c8185fa5114fd" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 39625 } } peers { permanent_uuid: "c0a24c924c484ac2be5dbec2b5bb45e1" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43753 } } peers { permanent_uuid: "4bed4a0e54674a99ba4661ddb1ce6395" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37037 } }
I20260925 22:44:53.038024 19321 leader_election.cc:288] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c767841753fa44429e8c8185fa5114fd (127.2.74.194:39625), c0a24c924c484ac2be5dbec2b5bb45e1 (127.2.74.195:43753)
W20260925 22:44:53.039110 18889 leader_election.cc:341] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c767841753fa44429e8c8185fa5114fd (127.2.74.194:39625): Not found: Tablet not found: ce32e77aacbe4a18925cc6a49ab7997b
W20260925 22:44:53.039317 18889 leader_election.cc:341] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c0a24c924c484ac2be5dbec2b5bb45e1 (127.2.74.195:43753): Not found: Tablet not found: ce32e77aacbe4a18925cc6a49ab7997b
I20260925 22:44:53.039381 18889 leader_election.cc:302] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bed4a0e54674a99ba4661ddb1ce6395; no voters: c0a24c924c484ac2be5dbec2b5bb45e1, c767841753fa44429e8c8185fa5114fd
I20260925 22:44:53.039551 19321 raft_consensus.cc:2726] T ce32e77aacbe4a18925cc6a49ab7997b P 4bed4a0e54674a99ba4661ddb1ce6395 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:44:53.315580 19322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.315095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.193:42685)
0925 22:44:53.315412 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:53.315480 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":72,"threads_started":1}
I20260925 22:44:53.488330 19323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.487894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34049 (local address 127.2.74.194:33083)
0925 22:44:53.488167 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:53.488231 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.254:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":77,"threads_started":1}
I20260925 22:44:53.671495 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 18831
I20260925 22:44:53.677686 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 19004
I20260925 22:44:53.683509 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 19139
2026-09-25T22:44:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59299 ms)
[----------] 30 tests from SecurityITest (246094 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:44:53 dist-test-slave-csnl krb5kdc[19326](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:44:53 dist-test-slave-csnl krb5kdc[19326](info): set up 2 sockets
Sep 25 22:44:53 dist-test-slave-csnl krb5kdc[19326](info): commencing operation
krb5kdc: starting...
W20260925 22:44:54.722887 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:44:54 dist-test-slave-csnl krb5kdc[19326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:44:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:44:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:44:54.854483 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:42009
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:37875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:42009
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:54.969775 19339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:54.970118 19339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:54.970165 19339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:54.974427 19339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:44:54.974520 19339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:54.974540 19339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:54.974557 19339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:44:54.974575 19339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:44:54.980404 19339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:37875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:42009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:42009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:44:54.981559 19339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:54.982610 19339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:54.991799 19345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:54.991809 19347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:54.991813 19344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:54.992110 19339 server_base.cc:1034] running on GCE node
I20260925 22:44:54.992884 19339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:54.994091 19339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:54.995261 19339 hybrid_clock.cc:648] HybridClock initialized: now 1790376294995231 us; error 46 us; skew 500 ppm
Sep 25 22:44:55 dist-test-slave-csnl krb5kdc[19326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376295, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:55.005522 19339 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:44:55.006587 19339 webserver.cc:460] Webserver started at http://127.2.74.254:46231/ using document root <none> and password file <none>
I20260925 22:44:55.007296 19339 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:55.007376 19339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:55.007565 19339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:55.009791 19339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "59cc6b3c2b92471d8425d1db3fa7765f"
format_stamp: "Formatted at 2026-09-25 22:44:55 on dist-test-slave-csnl"
server_key: "482c56d2f50d224efc0d40cf65098209"
server_key_iv: "f30ab6fd005c4cc62af20f7fa314eb07"
server_key_version: "encryptionkey@0"
I20260925 22:44:55.010387 19339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "59cc6b3c2b92471d8425d1db3fa7765f"
format_stamp: "Formatted at 2026-09-25 22:44:55 on dist-test-slave-csnl"
server_key: "482c56d2f50d224efc0d40cf65098209"
server_key_iv: "f30ab6fd005c4cc62af20f7fa314eb07"
server_key_version: "encryptionkey@0"
I20260925 22:44:55.014880 19339 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 22:44:55.018061 19354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:55.019786 19339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:55.019961 19339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "59cc6b3c2b92471d8425d1db3fa7765f"
format_stamp: "Formatted at 2026-09-25 22:44:55 on dist-test-slave-csnl"
server_key: "482c56d2f50d224efc0d40cf65098209"
server_key_iv: "f30ab6fd005c4cc62af20f7fa314eb07"
server_key_version: "encryptionkey@0"
I20260925 22:44:55.020103 19339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:55.047891 19339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:55.051600 19339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:55.051833 19339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:55.059653 19339 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:42009
I20260925 22:44:55.059662 19406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:42009 every 8 connection(s)
I20260925 22:44:55.060770 19339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:44:55.061764 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 19339
I20260925 22:44:55.061931 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:44:55.062222 2347 external_mini_cluster.cc:1428] Setting key 62067cf8df270864d6276ae54f23a823
I20260925 22:44:55.064602 19407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:55.072187 19407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59cc6b3c2b92471d8425d1db3fa7765f: Bootstrap starting.
Sep 25 22:44:55 dist-test-slave-csnl krb5kdc[19326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:44:55.075232 19407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59cc6b3c2b92471d8425d1db3fa7765f: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:55.076198 19407 log.cc:824] T 00000000000000000000000000000000 P 59cc6b3c2b92471d8425d1db3fa7765f: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:55.078486 19407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59cc6b3c2b92471d8425d1db3fa7765f: No bootstrap required, opened a new log
I20260925 22:44:55.079310 19410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.063714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56916 (local address 127.2.74.254:42009)
0925 22:44:55.064193 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:55.064220 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:55.064271 (+ 51us) server_negotiation.cc:407] Connection header received
0925 22:44:55.065179 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:55.065206 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:55.065667 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:55.066086 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:55.067047 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.068630 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.069711 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.070474 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.070648 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:55.073678 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:55.073686 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:55.073705 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:55.073748 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:55.076035 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:55.076568 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:55.076578 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:55.076589 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:55.076751 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:55.077076 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:55.077079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:55.077081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:55.077533 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:55.077752 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:55.078077 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 22:44:55.078388 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":129,"threads_started":1}
I20260925 22:44:55.082479 19407 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59cc6b3c2b92471d8425d1db3fa7765f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59cc6b3c2b92471d8425d1db3fa7765f" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 42009 } }
I20260925 22:44:55.082790 19407 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59cc6b3c2b92471d8425d1db3fa7765f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:55.082871 19407 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59cc6b3c2b92471d8425d1db3fa7765f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59cc6b3c2b92471d8425d1db3fa7765f, State: Initialized, Role: FOLLOWER
I20260925 22:44:55.083351 19407 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59cc6b3c2b92471d8425d1db3fa7765f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59cc6b3c2b92471d8425d1db3fa7765f" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 42009 } }
I20260925 22:44:55.083518 19407 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59cc6b3c2b92471d8425d1db3fa7765f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:44:55.083602 19407 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59cc6b3c2b92471d8425d1db3fa7765f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:44:55.083717 19407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59cc6b3c2b92471d8425d1db3fa7765f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:55.084738 19407 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59cc6b3c2b92471d8425d1db3fa7765f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59cc6b3c2b92471d8425d1db3fa7765f" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 42009 } }
I20260925 22:44:55.085117 19407 leader_election.cc:302] T 00000000000000000000000000000000 P 59cc6b3c2b92471d8425d1db3fa7765f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59cc6b3c2b92471d8425d1db3fa7765f; no voters:
I20260925 22:44:55.085467 19407 leader_election.cc:288] T 00000000000000000000000000000000 P 59cc6b3c2b92471d8425d1db3fa7765f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:44:55.085695 19412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59cc6b3c2b92471d8425d1db3fa7765f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:55.085952 19412 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59cc6b3c2b92471d8425d1db3fa7765f [term 1 LEADER]: Becoming Leader. State: Replica: 59cc6b3c2b92471d8425d1db3fa7765f, State: Running, Role: LEADER
I20260925 22:44:55.086338 19412 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59cc6b3c2b92471d8425d1db3fa7765f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59cc6b3c2b92471d8425d1db3fa7765f" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 42009 } }
I20260925 22:44:55.086907 19407 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59cc6b3c2b92471d8425d1db3fa7765f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:44:55.088363 19414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59cc6b3c2b92471d8425d1db3fa7765f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59cc6b3c2b92471d8425d1db3fa7765f. Latest consensus state: current_term: 1 leader_uuid: "59cc6b3c2b92471d8425d1db3fa7765f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59cc6b3c2b92471d8425d1db3fa7765f" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 42009 } } }
I20260925 22:44:55.088399 19413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59cc6b3c2b92471d8425d1db3fa7765f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59cc6b3c2b92471d8425d1db3fa7765f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59cc6b3c2b92471d8425d1db3fa7765f" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 42009 } } }
I20260925 22:44:55.088519 19414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59cc6b3c2b92471d8425d1db3fa7765f [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:55.088521 19413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59cc6b3c2b92471d8425d1db3fa7765f [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:55.088955 19423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:44:55.092476 19423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:44:55.098564 19423 catalog_manager.cc:1347] Generated new cluster ID: 164c4bfe902e47719aa1b16e00c8e1d1
I20260925 22:44:55.098658 19423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:44:55.106181 19423 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:44:55.107264 19423 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:44:55.114892 19423 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59cc6b3c2b92471d8425d1db3fa7765f: Generated new TSK 0
I20260925 22:44:55.115701 19423 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
I20260925 22:44:55.152485 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:0
--local_ip_for_outbound_sockets=127.2.74.193
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:42009
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:37875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:55.272182 19433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:55.272508 19433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:55.272555 19433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:55.276795 19433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:55.276876 19433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:55.276940 19433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:44:55.282811 19433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:37875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:42009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:44:55.283987 19433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:55.285020 19433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:55.296658 19439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:55.296720 19441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:55.296684 19433 server_base.cc:1034] running on GCE node
W20260925 22:44:55.296669 19438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:55.297256 19433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:55.298023 19433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:55.299224 19433 hybrid_clock.cc:648] HybridClock initialized: now 1790376295299204 us; error 34 us; skew 500 ppm
Sep 25 22:44:55 dist-test-slave-csnl krb5kdc[19326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376295, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:55.302151 19433 init.cc:377] Logged in from keytab as kudu/127.2.74.193@KRBTEST.COM (short username kudu)
I20260925 22:44:55.303217 19433 webserver.cc:460] Webserver started at http://127.2.74.193:42809/ using document root <none> and password file <none>
I20260925 22:44:55.303985 19433 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:55.304090 19433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:55.304312 19433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:55.306660 19433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-0/data/instance:
uuid: "a6c855a6dc3149d2a9f5e670a124b144"
format_stamp: "Formatted at 2026-09-25 22:44:55 on dist-test-slave-csnl"
server_key: "ba66e1f9babec47489dd297d81dd71c7"
server_key_iv: "5e11d42cdd5579f2680515fa9948eb4e"
server_key_version: "encryptionkey@0"
I20260925 22:44:55.307358 19433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance:
uuid: "a6c855a6dc3149d2a9f5e670a124b144"
format_stamp: "Formatted at 2026-09-25 22:44:55 on dist-test-slave-csnl"
server_key: "ba66e1f9babec47489dd297d81dd71c7"
server_key_iv: "5e11d42cdd5579f2680515fa9948eb4e"
server_key_version: "encryptionkey@0"
I20260925 22:44:55.311506 19433 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:44:55.314270 19448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:55.315760 19433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:55.315941 19433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "a6c855a6dc3149d2a9f5e670a124b144"
format_stamp: "Formatted at 2026-09-25 22:44:55 on dist-test-slave-csnl"
server_key: "ba66e1f9babec47489dd297d81dd71c7"
server_key_iv: "5e11d42cdd5579f2680515fa9948eb4e"
server_key_version: "encryptionkey@0"
I20260925 22:44:55.316083 19433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:55.361905 19433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:55.365494 19433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:55.365759 19433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:55.366472 19433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:55.367440 19433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:55.367520 19433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:55.367597 19433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:55.367642 19433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:55.377180 19433 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:46569
I20260925 22:44:55.377202 19561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:46569 every 8 connection(s)
I20260925 22:44:55.378198 19433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:44:55.380566 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 19433
I20260925 22:44:55.380719 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance
I20260925 22:44:55.381033 2347 external_mini_cluster.cc:1428] Setting key 904ccbd39094ee5ea3f70357abf75bed
Sep 25 22:44:55 dist-test-slave-csnl krb5kdc[19326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376295, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.74.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.194@KRBTEST.COM" created.
I20260925 22:44:55.394448 19410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.380399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:34553 (local address 127.2.74.254:42009)
0925 22:44:55.380570 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:55.380575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:55.381679 (+ 1104us) server_negotiation.cc:407] Connection header received
0925 22:44:55.383070 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:55.383076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:55.383142 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:55.383237 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:55.384544 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.385096 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.386346 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.386685 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.386808 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:55.390176 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:55.390180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:55.390182 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:55.390211 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:55.392032 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:55.392703 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:55.392707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:55.392709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:55.392804 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:55.393207 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:55.393211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:55.393213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:55.393429 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:55.393551 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:55.394121 (+ 570us) server_negotiation.cc:299] Negotiation successful
0925 22:44:55.394274 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 22:44:55.395627 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.380939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:42009 (local address 127.2.74.193:34553)
0925 22:44:55.381504 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:55.381564 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:55.382796 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:55.383409 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:55.383417 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:55.383805 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:55.384356 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:55.384372 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.385247 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.385252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:55.386195 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:55.386205 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.386863 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.386868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:55.387130 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:55.387887 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:55.387916 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:55.389936 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:55.392208 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:55.392216 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:55.392238 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:55.392574 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:55.392909 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:55.392912 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:55.392914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:55.393099 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:55.393552 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:55.393563 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:55.393856 (+ 293us) client_negotiation.cc:769] Sending connection context
0925 22:44:55.394067 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 22:44:55.394464 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":191,"threads_started":1}
I20260925 22:44:55.397095 19562 heartbeater.cc:347] Connected to a master server at 127.2.74.254:42009
I20260925 22:44:55.397415 19562 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:55.398106 19562 heartbeater.cc:511] Master 127.2.74.254:42009 requested a full tablet report, sending...
I20260925 22:44:55.400322 19371 ts_manager.cc:194] Registered new tserver with Master: a6c855a6dc3149d2a9f5e670a124b144 (127.2.74.193:46569)
I20260925 22:44:55.402132 19371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.193@KRBTEST.COM'} at 127.2.74.193:34553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
I20260925 22:44:55.410516 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:0
--local_ip_for_outbound_sockets=127.2.74.194
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:42009
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:37875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:55.554925 19567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:55.555294 19567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:55.555367 19567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:55.559823 19567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:55.559914 19567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:55.560005 19567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:44:55.565999 19567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:37875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:42009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:44:55.567307 19567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:55.568432 19567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:55.580879 19575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:55.580991 19573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:55.580883 19572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:55.581425 19567 server_base.cc:1034] running on GCE node
I20260925 22:44:55.581897 19567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:55.582695 19567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:55.583900 19567 hybrid_clock.cc:648] HybridClock initialized: now 1790376295583871 us; error 73 us; skew 500 ppm
Sep 25 22:44:55 dist-test-slave-csnl krb5kdc[19326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376295, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:55.587049 19567 init.cc:377] Logged in from keytab as kudu/127.2.74.194@KRBTEST.COM (short username kudu)
I20260925 22:44:55.588446 19567 webserver.cc:460] Webserver started at http://127.2.74.194:38187/ using document root <none> and password file <none>
I20260925 22:44:55.589190 19567 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:55.589264 19567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:55.589556 19567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:55.591892 19567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-1/data/instance:
uuid: "3f0dd29964e34637a44c270797a5179f"
format_stamp: "Formatted at 2026-09-25 22:44:55 on dist-test-slave-csnl"
server_key: "019e7c88208026239a8e3349b802a990"
server_key_iv: "b7a4aa385a487e94720c7b74c8f5a7f5"
server_key_version: "encryptionkey@0"
I20260925 22:44:55.592528 19567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance:
uuid: "3f0dd29964e34637a44c270797a5179f"
format_stamp: "Formatted at 2026-09-25 22:44:55 on dist-test-slave-csnl"
server_key: "019e7c88208026239a8e3349b802a990"
server_key_iv: "b7a4aa385a487e94720c7b74c8f5a7f5"
server_key_version: "encryptionkey@0"
I20260925 22:44:55.597122 19567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:44:55.600382 19582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:55.602044 19567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:55.602233 19567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "3f0dd29964e34637a44c270797a5179f"
format_stamp: "Formatted at 2026-09-25 22:44:55 on dist-test-slave-csnl"
server_key: "019e7c88208026239a8e3349b802a990"
server_key_iv: "b7a4aa385a487e94720c7b74c8f5a7f5"
server_key_version: "encryptionkey@0"
I20260925 22:44:55.602386 19567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:55.615732 19567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:55.619215 19567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:55.619459 19567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:55.620153 19567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:55.621054 19567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:55.621129 19567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:55.621202 19567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:55.621244 19567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:55.630710 19567 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:38541
I20260925 22:44:55.630733 19695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:38541 every 8 connection(s)
I20260925 22:44:55.631845 19567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:44:55.638336 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 19567
I20260925 22:44:55.638458 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance
I20260925 22:44:55.638754 2347 external_mini_cluster.cc:1428] Setting key 2bb456a20aaa0c09b0a41963922883ba
Sep 25 22:44:55 dist-test-slave-csnl krb5kdc[19326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376295, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:44:55.645788 19410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.633783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:60879 (local address 127.2.74.254:42009)
0925 22:44:55.633999 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:55.634004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:55.634900 (+ 896us) server_negotiation.cc:407] Connection header received
0925 22:44:55.635858 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:55.635861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:55.635915 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:55.635992 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:55.637176 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.637692 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.639028 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.639332 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.639470 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:55.641996 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:55.642005 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:55.642007 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:55.642036 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:55.643510 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:55.644108 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:55.644112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:55.644114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:55.644204 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:55.644586 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:55.644591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:55.644593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:55.644800 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:55.644925 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:55.645478 (+ 553us) server_negotiation.cc:299] Negotiation successful
0925 22:44:55.645619 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260925 22:44:55.646792 19698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.634218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:42009 (local address 127.2.74.194:60879)
0925 22:44:55.634718 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:55.634765 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:55.635649 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:55.636162 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:55.636169 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:55.636547 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:55.637040 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:55.637054 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.637808 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.637811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:55.638870 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:55.638880 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.639518 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.639522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:55.639717 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:55.640284 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:55.640301 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:55.641797 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:55.643658 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:55.643664 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:55.643684 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:55.643987 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:55.644307 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:55.644311 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:55.644313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:55.644481 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:55.644926 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:55.644935 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:55.645217 (+ 282us) client_negotiation.cc:769] Sending connection context
0925 22:44:55.645431 (+ 214us) client_negotiation.cc:240] Negotiation successful
0925 22:44:55.645797 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":134,"threads_started":1}
I20260925 22:44:55.648166 19696 heartbeater.cc:347] Connected to a master server at 127.2.74.254:42009
I20260925 22:44:55.648420 19696 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.74.195@KRBTEST.COM; defaulting to no policy
I20260925 22:44:55.648998 19696 heartbeater.cc:511] Master 127.2.74.254:42009 requested a full tablet report, sending...
I20260925 22:44:55.650127 19371 ts_manager.cc:194] Registered new tserver with Master: 3f0dd29964e34637a44c270797a5179f (127.2.74.194:38541)
I20260925 22:44:55.650804 19371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.194@KRBTEST.COM'} at 127.2.74.194:60879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
I20260925 22:44:55.667315 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:0
--local_ip_for_outbound_sockets=127.2.74.195
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:42009
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:37875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:55.785976 19701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:55.786329 19701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:55.786405 19701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:55.791134 19701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:55.791209 19701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:55.791311 19701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:44:55.797070 19701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:37875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:42009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:44:55.798223 19701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:55.799386 19701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:55.809767 19709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:55.809792 19706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:55.809854 19701 server_base.cc:1034] running on GCE node
W20260925 22:44:55.809799 19707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:55.810503 19701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:55.811265 19701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:55.812450 19701 hybrid_clock.cc:648] HybridClock initialized: now 1790376295812433 us; error 35 us; skew 500 ppm
Sep 25 22:44:55 dist-test-slave-csnl krb5kdc[19326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376295, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:55.815603 19701 init.cc:377] Logged in from keytab as kudu/127.2.74.195@KRBTEST.COM (short username kudu)
I20260925 22:44:55.816566 19701 webserver.cc:460] Webserver started at http://127.2.74.195:42717/ using document root <none> and password file <none>
I20260925 22:44:55.817135 19701 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:55.817200 19701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:55.817376 19701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:55.819617 19701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-2/data/instance:
uuid: "cd0116b8b49f4e49951ff76430e38780"
format_stamp: "Formatted at 2026-09-25 22:44:55 on dist-test-slave-csnl"
server_key: "e60b080536212c6e59ce901c5a57a94e"
server_key_iv: "bca13e6aeabb30a533995965f0ddc11c"
server_key_version: "encryptionkey@0"
I20260925 22:44:55.820183 19701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance:
uuid: "cd0116b8b49f4e49951ff76430e38780"
format_stamp: "Formatted at 2026-09-25 22:44:55 on dist-test-slave-csnl"
server_key: "e60b080536212c6e59ce901c5a57a94e"
server_key_iv: "bca13e6aeabb30a533995965f0ddc11c"
server_key_version: "encryptionkey@0"
I20260925 22:44:55.824409 19701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 22:44:55.827229 19716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:55.828586 19701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:44:55.828694 19701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "cd0116b8b49f4e49951ff76430e38780"
format_stamp: "Formatted at 2026-09-25 22:44:55 on dist-test-slave-csnl"
server_key: "e60b080536212c6e59ce901c5a57a94e"
server_key_iv: "bca13e6aeabb30a533995965f0ddc11c"
server_key_version: "encryptionkey@0"
I20260925 22:44:55.828792 19701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:55.865537 19701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:55.869701 19701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:55.869908 19701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:55.870484 19701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:55.871459 19701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:55.871510 19701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:55.871554 19701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:55.871569 19701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:55.880950 19701 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:43459
I20260925 22:44:55.881012 19829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:43459 every 8 connection(s)
I20260925 22:44:55.882002 19701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:44:55.884930 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 19701
I20260925 22:44:55.885103 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance
I20260925 22:44:55.885432 2347 external_mini_cluster.cc:1428] Setting key cc21222f1c0b064473e4ba36707d8364
Sep 25 22:44:55 dist-test-slave-csnl krb5kdc[19326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376295, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:44:55.898078 19410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.884225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:51689 (local address 127.2.74.254:42009)
0925 22:44:55.884398 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:55.884404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:55.885998 (+ 1594us) server_negotiation.cc:407] Connection header received
0925 22:44:55.887305 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:55.887308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:55.887357 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:55.887468 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:55.889025 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.889576 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.890901 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.891210 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.891305 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:55.894180 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:55.894185 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:55.894188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:55.894223 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:55.895930 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:55.896508 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:55.896511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:55.896513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:55.896619 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:55.896966 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:55.896969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:55.896970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:55.897172 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:55.897256 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:55.897816 (+ 560us) server_negotiation.cc:299] Negotiation successful
0925 22:44:55.897946 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 22:44:55.899230 19832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.884779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:42009 (local address 127.2.74.195:51689)
0925 22:44:55.885727 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:55.885840 (+ 113us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:55.887043 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:55.887700 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:55.887711 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:55.888266 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:55.888873 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:55.888888 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.889757 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.889760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:55.890753 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:55.890764 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.891369 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.891372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:55.891546 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:55.892192 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:55.892211 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:55.893952 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:55.896058 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:55.896064 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:55.896081 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:55.896381 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:55.896721 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:55.896723 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:55.896725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:55.896874 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:55.897268 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:55.897274 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:55.897564 (+ 290us) client_negotiation.cc:769] Sending connection context
0925 22:44:55.897744 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 22:44:55.898044 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":179,"threads_started":1}
I20260925 22:44:55.900631 19830 heartbeater.cc:347] Connected to a master server at 127.2.74.254:42009
I20260925 22:44:55.900954 19830 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:55.901531 19830 heartbeater.cc:511] Master 127.2.74.254:42009 requested a full tablet report, sending...
I20260925 22:44:55.902808 19371 ts_manager.cc:194] Registered new tserver with Master: cd0116b8b49f4e49951ff76430e38780 (127.2.74.195:43459)
I20260925 22:44:55.903499 19371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.195@KRBTEST.COM'} at 127.2.74.195:51689
I20260925 22:44:55.912461 2347 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:44:55.923000 19410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.914695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56922 (local address 127.2.74.254:42009)
0925 22:44:55.914866 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:55.914871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:55.915036 (+ 165us) server_negotiation.cc:407] Connection header received
0925 22:44:55.915186 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:55.915188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:55.915233 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:55.915301 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:55.916010 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.916536 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.917362 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.917581 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.917658 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:55.919209 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:55.919212 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:55.919215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:55.919242 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:55.920679 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:55.921592 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:55.921595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:55.921597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:55.921685 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:55.922005 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:55.922014 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:55.922015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:55.922301 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:55.922439 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:55.922651 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 22:44:55.922766 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 22:44:55.926626 19371 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:44:55.929939 19371 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:44:55.945562 19846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.940445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58720 (local address 127.2.74.195:43459)
0925 22:44:55.940784 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:55.940795 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:55.940820 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:44:55.940910 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:55.940917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:55.941103 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:55.941263 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:55.941486 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.942650 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.944185 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.944807 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.944946 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:55.945138 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:55.945276 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 22:44:55.945392 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":142,"threads_started":1}
I20260925 22:44:55.945552 19844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.940007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:43459 (local address 127.0.0.1:58720)
0925 22:44:55.940465 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:55.940477 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:55.940566 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:55.941246 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:55.941249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:55.941268 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:55.941365 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:55.941372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.942801 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.942804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:55.944057 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:55.944065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.944944 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.944948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:55.945065 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:55.945108 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 22:44:55.945245 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 22:44:55.945394 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":80,"thread_start_us":96,"threads_started":1}
I20260925 22:44:55.945873 19842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.939347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41136 (local address 127.2.74.194:38541)
0925 22:44:55.940239 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:55.940251 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:55.940276 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:44:55.940388 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:55.940401 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:55.940643 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:55.941218 (+ 575us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:55.941392 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.942184 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.943979 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.944869 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.944979 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:55.945126 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:55.945624 (+ 498us) server_negotiation.cc:299] Negotiation successful
0925 22:44:55.945724 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":159,"threads_started":1}
I20260925 22:44:55.946020 19843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.939544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:46569 (local address 127.0.0.1:46506)
0925 22:44:55.940150 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:55.940163 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:55.940253 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:55.941309 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:55.941315 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:55.941334 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:55.941477 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:55.941484 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.943054 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.943057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:55.944153 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:55.944166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.945293 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.945296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:55.945411 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:55.945432 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:44:55.945472 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 22:44:55.945528 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":2,"thread_start_us":80,"threads_started":1}
I20260925 22:44:55.946820 19845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.940048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46506 (local address 127.2.74.193:46569)
0925 22:44:55.940898 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:55.940904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:55.940922 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:44:55.941011 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:55.941016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:55.941165 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:55.941641 (+ 476us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:55.941790 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.942903 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.944322 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.945166 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.946351 (+ 1185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:55.946496 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:55.946570 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 22:44:55.946672 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":113,"threads_started":1}
I20260925 22:44:55.949242 19764 tablet_service.cc:1467] Processing CreateTablet for tablet d03ef7526fdc46f79ad560503c4d71d4 (DEFAULT_TABLE table=test-table [id=fcd57480d92e440da691525a601d85de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:55.950120 19841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.939218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:38541 (local address 127.0.0.1:41136)
0925 22:44:55.939595 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:55.939646 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:55.939772 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:55.940832 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:55.940838 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:55.940862 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:55.941050 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:55.941057 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.942315 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.942320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:55.943831 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:55.943842 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.945195 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.945202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:55.945315 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:55.945342 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 22:44:55.945411 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 22:44:55.945511 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":2,"thread_start_us":118,"threads_started":1}
I20260925 22:44:55.949682 19496 tablet_service.cc:1467] Processing CreateTablet for tablet d03ef7526fdc46f79ad560503c4d71d4 (DEFAULT_TABLE table=test-table [id=fcd57480d92e440da691525a601d85de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:55.950553 19764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d03ef7526fdc46f79ad560503c4d71d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:55.950762 19496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d03ef7526fdc46f79ad560503c4d71d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:55.953140 19630 tablet_service.cc:1467] Processing CreateTablet for tablet d03ef7526fdc46f79ad560503c4d71d4 (DEFAULT_TABLE table=test-table [id=fcd57480d92e440da691525a601d85de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:55.954154 19630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d03ef7526fdc46f79ad560503c4d71d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:55.958570 19847 tablet_bootstrap.cc:492] T d03ef7526fdc46f79ad560503c4d71d4 P a6c855a6dc3149d2a9f5e670a124b144: Bootstrap starting.
I20260925 22:44:55.959112 19848 tablet_bootstrap.cc:492] T d03ef7526fdc46f79ad560503c4d71d4 P cd0116b8b49f4e49951ff76430e38780: Bootstrap starting.
I20260925 22:44:55.961328 19849 tablet_bootstrap.cc:492] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f: Bootstrap starting.
I20260925 22:44:55.961766 19848 tablet_bootstrap.cc:654] T d03ef7526fdc46f79ad560503c4d71d4 P cd0116b8b49f4e49951ff76430e38780: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:55.963222 19848 log.cc:824] T d03ef7526fdc46f79ad560503c4d71d4 P cd0116b8b49f4e49951ff76430e38780: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:55.964229 19849 tablet_bootstrap.cc:654] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:55.964386 19847 tablet_bootstrap.cc:654] T d03ef7526fdc46f79ad560503c4d71d4 P a6c855a6dc3149d2a9f5e670a124b144: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:55.965044 19849 log.cc:824] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:55.965199 19847 log.cc:824] T d03ef7526fdc46f79ad560503c4d71d4 P a6c855a6dc3149d2a9f5e670a124b144: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:55.965777 19848 tablet_bootstrap.cc:492] T d03ef7526fdc46f79ad560503c4d71d4 P cd0116b8b49f4e49951ff76430e38780: No bootstrap required, opened a new log
I20260925 22:44:55.966022 19848 ts_tablet_manager.cc:1397] T d03ef7526fdc46f79ad560503c4d71d4 P cd0116b8b49f4e49951ff76430e38780: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:44:55.967427 19849 tablet_bootstrap.cc:492] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f: No bootstrap required, opened a new log
I20260925 22:44:55.967427 19847 tablet_bootstrap.cc:492] T d03ef7526fdc46f79ad560503c4d71d4 P a6c855a6dc3149d2a9f5e670a124b144: No bootstrap required, opened a new log
I20260925 22:44:55.967646 19847 ts_tablet_manager.cc:1397] T d03ef7526fdc46f79ad560503c4d71d4 P a6c855a6dc3149d2a9f5e670a124b144: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.004s
I20260925 22:44:55.967646 19849 ts_tablet_manager.cc:1397] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:44:55.969651 19848 raft_consensus.cc:348] T d03ef7526fdc46f79ad560503c4d71d4 P cd0116b8b49f4e49951ff76430e38780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f0dd29964e34637a44c270797a5179f" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38541 } } peers { permanent_uuid: "cd0116b8b49f4e49951ff76430e38780" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43459 } } peers { permanent_uuid: "a6c855a6dc3149d2a9f5e670a124b144" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 46569 } }
I20260925 22:44:55.969991 19848 raft_consensus.cc:374] T d03ef7526fdc46f79ad560503c4d71d4 P cd0116b8b49f4e49951ff76430e38780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:55.970068 19848 raft_consensus.cc:729] T d03ef7526fdc46f79ad560503c4d71d4 P cd0116b8b49f4e49951ff76430e38780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd0116b8b49f4e49951ff76430e38780, State: Initialized, Role: FOLLOWER
I20260925 22:44:55.970587 19848 consensus_queue.cc:260] T d03ef7526fdc46f79ad560503c4d71d4 P cd0116b8b49f4e49951ff76430e38780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f0dd29964e34637a44c270797a5179f" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38541 } } peers { permanent_uuid: "cd0116b8b49f4e49951ff76430e38780" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43459 } } peers { permanent_uuid: "a6c855a6dc3149d2a9f5e670a124b144" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 46569 } }
I20260925 22:44:55.971341 19830 heartbeater.cc:503] Master 127.2.74.254:42009 was elected leader, sending a full tablet report...
I20260925 22:44:55.971767 19848 ts_tablet_manager.cc:1428] T d03ef7526fdc46f79ad560503c4d71d4 P cd0116b8b49f4e49951ff76430e38780: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260925 22:44:55.971904 19849 raft_consensus.cc:348] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f0dd29964e34637a44c270797a5179f" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38541 } } peers { permanent_uuid: "cd0116b8b49f4e49951ff76430e38780" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43459 } } peers { permanent_uuid: "a6c855a6dc3149d2a9f5e670a124b144" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 46569 } }
I20260925 22:44:55.971904 19847 raft_consensus.cc:348] T d03ef7526fdc46f79ad560503c4d71d4 P a6c855a6dc3149d2a9f5e670a124b144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f0dd29964e34637a44c270797a5179f" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38541 } } peers { permanent_uuid: "cd0116b8b49f4e49951ff76430e38780" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43459 } } peers { permanent_uuid: "a6c855a6dc3149d2a9f5e670a124b144" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 46569 } }
I20260925 22:44:55.972209 19847 raft_consensus.cc:374] T d03ef7526fdc46f79ad560503c4d71d4 P a6c855a6dc3149d2a9f5e670a124b144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:55.972209 19849 raft_consensus.cc:374] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:55.972283 19849 raft_consensus.cc:729] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f0dd29964e34637a44c270797a5179f, State: Initialized, Role: FOLLOWER
I20260925 22:44:55.972326 19847 raft_consensus.cc:729] T d03ef7526fdc46f79ad560503c4d71d4 P a6c855a6dc3149d2a9f5e670a124b144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6c855a6dc3149d2a9f5e670a124b144, State: Initialized, Role: FOLLOWER
I20260925 22:44:55.972828 19849 consensus_queue.cc:260] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f0dd29964e34637a44c270797a5179f" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38541 } } peers { permanent_uuid: "cd0116b8b49f4e49951ff76430e38780" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43459 } } peers { permanent_uuid: "a6c855a6dc3149d2a9f5e670a124b144" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 46569 } }
I20260925 22:44:55.972828 19847 consensus_queue.cc:260] T d03ef7526fdc46f79ad560503c4d71d4 P a6c855a6dc3149d2a9f5e670a124b144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f0dd29964e34637a44c270797a5179f" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38541 } } peers { permanent_uuid: "cd0116b8b49f4e49951ff76430e38780" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43459 } } peers { permanent_uuid: "a6c855a6dc3149d2a9f5e670a124b144" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 46569 } }
I20260925 22:44:55.973938 19847 ts_tablet_manager.cc:1428] T d03ef7526fdc46f79ad560503c4d71d4 P a6c855a6dc3149d2a9f5e670a124b144: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:44:55.973932 19849 ts_tablet_manager.cc:1428] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:44:55.974778 19696 heartbeater.cc:503] Master 127.2.74.254:42009 was elected leader, sending a full tablet report...
I20260925 22:44:55.975852 19562 heartbeater.cc:503] Master 127.2.74.254:42009 was elected leader, sending a full tablet report...
I20260925 22:44:56.003602 19855 raft_consensus.cc:482] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:44:56.003835 19855 raft_consensus.cc:504] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f0dd29964e34637a44c270797a5179f" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38541 } } peers { permanent_uuid: "cd0116b8b49f4e49951ff76430e38780" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43459 } } peers { permanent_uuid: "a6c855a6dc3149d2a9f5e670a124b144" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 46569 } }
I20260925 22:44:56.005136 19855 leader_election.cc:288] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd0116b8b49f4e49951ff76430e38780 (127.2.74.195:43459), a6c855a6dc3149d2a9f5e670a124b144 (127.2.74.193:46569)
I20260925 22:44:56.009778 19698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:56.005441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:46569 (local address 127.2.74.194:50621)
0925 22:44:56.005671 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:56.005688 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:56.005790 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:56.006155 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:56.006158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:56.006178 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:56.006343 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:56.006373 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.007277 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.007280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:56.008316 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:56.008325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.009285 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.009290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:56.009423 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:56.009451 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 22:44:56.009540 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 22:44:56.009622 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260925 22:44:56.009826 19845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:56.005595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:50621 (local address 127.2.74.193:46569)
0925 22:44:56.005765 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:56.005769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:56.005784 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:44:56.005948 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:56.005951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:56.006041 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:56.006140 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:56.006507 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.007143 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.008451 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.009142 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.009242 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:56.009290 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:56.009596 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 22:44:56.009674 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 22:44:56.010516 19856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:56.005447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:43459 (local address 127.2.74.194:36145)
0925 22:44:56.006194 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:56.006206 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:56.006291 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:56.006794 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:56.006797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:56.006818 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:56.006921 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:56.006926 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.007926 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.007929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:56.008987 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:56.008999 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.010110 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.010116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:56.010216 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:56.010242 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 22:44:56.010336 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 22:44:56.010401 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":351,"thread_start_us":179,"threads_started":1}
I20260925 22:44:56.010573 19846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:56.005636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:36145 (local address 127.2.74.195:43459)
0925 22:44:56.005781 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:56.005785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:56.006273 (+ 488us) server_negotiation.cc:407] Connection header received
0925 22:44:56.006471 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:56.006476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:56.006539 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:56.006608 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:56.007057 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.007817 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.009148 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.009802 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.009961 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:56.010019 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:56.010374 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 22:44:56.010460 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:44:56.010679 19516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d03ef7526fdc46f79ad560503c4d71d4" candidate_uuid: "3f0dd29964e34637a44c270797a5179f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6c855a6dc3149d2a9f5e670a124b144" is_pre_election: true
I20260925 22:44:56.011108 19516 raft_consensus.cc:2454] T d03ef7526fdc46f79ad560503c4d71d4 P a6c855a6dc3149d2a9f5e670a124b144 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f0dd29964e34637a44c270797a5179f in term 0.
I20260925 22:44:56.011214 19783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d03ef7526fdc46f79ad560503c4d71d4" candidate_uuid: "3f0dd29964e34637a44c270797a5179f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd0116b8b49f4e49951ff76430e38780" is_pre_election: true
I20260925 22:44:56.011552 19783 raft_consensus.cc:2454] T d03ef7526fdc46f79ad560503c4d71d4 P cd0116b8b49f4e49951ff76430e38780 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f0dd29964e34637a44c270797a5179f in term 0.
I20260925 22:44:56.011691 19586 leader_election.cc:302] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f0dd29964e34637a44c270797a5179f, a6c855a6dc3149d2a9f5e670a124b144; no voters:
I20260925 22:44:56.011994 19855 raft_consensus.cc:2781] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:44:56.012084 19855 raft_consensus.cc:482] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:44:56.012127 19855 raft_consensus.cc:3037] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:56.013262 19855 raft_consensus.cc:504] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f0dd29964e34637a44c270797a5179f" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38541 } } peers { permanent_uuid: "cd0116b8b49f4e49951ff76430e38780" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43459 } } peers { permanent_uuid: "a6c855a6dc3149d2a9f5e670a124b144" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 46569 } }
I20260925 22:44:56.013636 19855 leader_election.cc:288] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f [CANDIDATE]: Term 1 election: Requested vote from peers cd0116b8b49f4e49951ff76430e38780 (127.2.74.195:43459), a6c855a6dc3149d2a9f5e670a124b144 (127.2.74.193:46569)
I20260925 22:44:56.014026 19783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d03ef7526fdc46f79ad560503c4d71d4" candidate_uuid: "3f0dd29964e34637a44c270797a5179f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd0116b8b49f4e49951ff76430e38780"
I20260925 22:44:56.014045 19516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d03ef7526fdc46f79ad560503c4d71d4" candidate_uuid: "3f0dd29964e34637a44c270797a5179f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6c855a6dc3149d2a9f5e670a124b144"
I20260925 22:44:56.014189 19516 raft_consensus.cc:3037] T d03ef7526fdc46f79ad560503c4d71d4 P a6c855a6dc3149d2a9f5e670a124b144 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:56.014187 19783 raft_consensus.cc:3037] T d03ef7526fdc46f79ad560503c4d71d4 P cd0116b8b49f4e49951ff76430e38780 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:56.015439 19783 raft_consensus.cc:2454] T d03ef7526fdc46f79ad560503c4d71d4 P cd0116b8b49f4e49951ff76430e38780 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f0dd29964e34637a44c270797a5179f in term 1.
I20260925 22:44:56.015499 19516 raft_consensus.cc:2454] T d03ef7526fdc46f79ad560503c4d71d4 P a6c855a6dc3149d2a9f5e670a124b144 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f0dd29964e34637a44c270797a5179f in term 1.
I20260925 22:44:56.015848 19585 leader_election.cc:302] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f0dd29964e34637a44c270797a5179f, cd0116b8b49f4e49951ff76430e38780; no voters:
I20260925 22:44:56.016057 19855 raft_consensus.cc:2781] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:56.016350 19855 raft_consensus.cc:686] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f [term 1 LEADER]: Becoming Leader. State: Replica: 3f0dd29964e34637a44c270797a5179f, State: Running, Role: LEADER
I20260925 22:44:56.016762 19855 consensus_queue.cc:237] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f0dd29964e34637a44c270797a5179f" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38541 } } peers { permanent_uuid: "cd0116b8b49f4e49951ff76430e38780" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43459 } } peers { permanent_uuid: "a6c855a6dc3149d2a9f5e670a124b144" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 46569 } }
I20260925 22:44:56.020538 19370 catalog_manager.cc:5495] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f0dd29964e34637a44c270797a5179f (127.2.74.194). New cstate: current_term: 1 leader_uuid: "3f0dd29964e34637a44c270797a5179f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f0dd29964e34637a44c270797a5179f" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 38541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd0116b8b49f4e49951ff76430e38780" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6c855a6dc3149d2a9f5e670a124b144" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 46569 } health_report { overall_health: UNKNOWN } } }
I20260925 22:44:56.046013 19842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:56.042291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41142 (local address 127.2.74.194:38541)
0925 22:44:56.042453 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:56.042457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:56.042472 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:44:56.042620 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:56.042623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:56.042680 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:56.042806 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:44:56.043077 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.043578 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.044473 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.044695 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.044807 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:56.045135 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:44:56.045584 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:44:56.045722 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:56.045788 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 22:44:56.045855 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 22:44:56.052294 19783 raft_consensus.cc:1261] T d03ef7526fdc46f79ad560503c4d71d4 P cd0116b8b49f4e49951ff76430e38780 [term 1 FOLLOWER]: Refusing update from remote peer 3f0dd29964e34637a44c270797a5179f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:56.052294 19516 raft_consensus.cc:1261] T d03ef7526fdc46f79ad560503c4d71d4 P a6c855a6dc3149d2a9f5e670a124b144 [term 1 FOLLOWER]: Refusing update from remote peer 3f0dd29964e34637a44c270797a5179f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:56.053249 19857 consensus_queue.cc:1035] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6c855a6dc3149d2a9f5e670a124b144" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 46569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:56.053489 19855 consensus_queue.cc:1035] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd0116b8b49f4e49951ff76430e38780" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:56.062877 19861 mvcc.cc:205] Tried to move back new op lower bound from 7333381308621623296 to 7333381308488216576. Current Snapshot: MvccSnapshot[applied={T|T < 7333381308621623296}]
I20260925 22:44:56.062999 19862 mvcc.cc:205] Tried to move back new op lower bound from 7333381308621623296 to 7333381308488216576. Current Snapshot: MvccSnapshot[applied={T|T < 7333381308621623296}]
I20260925 22:44:56.073208 19370 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:56.073417 19370 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:56.076033 19370 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59cc6b3c2b92471d8425d1db3fa7765f: Sending DeleteTablet for 3 replicas of tablet d03ef7526fdc46f79ad560503c4d71d4
I20260925 22:44:56.077054 19764 tablet_service.cc:1514] Processing DeleteTablet for tablet d03ef7526fdc46f79ad560503c4d71d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:56 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:58720
I20260925 22:44:56.077056 19630 tablet_service.cc:1514] Processing DeleteTablet for tablet d03ef7526fdc46f79ad560503c4d71d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:56 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:41136
I20260925 22:44:56.077090 19496 tablet_service.cc:1514] Processing DeleteTablet for tablet d03ef7526fdc46f79ad560503c4d71d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:56 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:46506
I20260925 22:44:56.077678 19870 tablet_replica.cc:330] T d03ef7526fdc46f79ad560503c4d71d4 P cd0116b8b49f4e49951ff76430e38780: stopping tablet replica
I20260925 22:44:56.078083 19870 raft_consensus.cc:2229] T d03ef7526fdc46f79ad560503c4d71d4 P cd0116b8b49f4e49951ff76430e38780 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:56.078382 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 19433
I20260925 22:44:56.078467 19870 raft_consensus.cc:2258] T d03ef7526fdc46f79ad560503c4d71d4 P cd0116b8b49f4e49951ff76430e38780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:56.080086 19871 tablet_replica.cc:330] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f: stopping tablet replica
I20260925 22:44:56.080469 19871 raft_consensus.cc:2229] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:44:56.080981 19871 raft_consensus.cc:2258] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:56.082835 19871 ts_tablet_manager.cc:1905] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:56.082932 19870 ts_tablet_manager.cc:1905] T d03ef7526fdc46f79ad560503c4d71d4 P cd0116b8b49f4e49951ff76430e38780: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:56.085613 19871 ts_tablet_manager.cc:1918] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:44:56.085773 19871 log.cc:1196] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-1/wal/wals/d03ef7526fdc46f79ad560503c4d71d4
I20260925 22:44:56.085907 19870 ts_tablet_manager.cc:1918] T d03ef7526fdc46f79ad560503c4d71d4 P cd0116b8b49f4e49951ff76430e38780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:44:56.086117 19870 log.cc:1196] T d03ef7526fdc46f79ad560503c4d71d4 P cd0116b8b49f4e49951ff76430e38780: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376047580784-2347-0/minicluster-data/ts-2/wal/wals/d03ef7526fdc46f79ad560503c4d71d4
I20260925 22:44:56.086155 19871 ts_tablet_manager.cc:1939] T d03ef7526fdc46f79ad560503c4d71d4 P 3f0dd29964e34637a44c270797a5179f: Deleting consensus metadata
I20260925 22:44:56.086561 19870 ts_tablet_manager.cc:1939] T d03ef7526fdc46f79ad560503c4d71d4 P cd0116b8b49f4e49951ff76430e38780: Deleting consensus metadata
I20260925 22:44:56.087911 19357 catalog_manager.cc:4841] TS 3f0dd29964e34637a44c270797a5179f (127.2.74.194:38541): tablet d03ef7526fdc46f79ad560503c4d71d4 (table test-table [id=fcd57480d92e440da691525a601d85de]) successfully deleted
I20260925 22:44:56.088184 19357 catalog_manager.cc:4841] TS cd0116b8b49f4e49951ff76430e38780 (127.2.74.195:43459): tablet d03ef7526fdc46f79ad560503c4d71d4 (table test-table [id=fcd57480d92e440da691525a601d85de]) successfully deleted
I20260925 22:44:56.091439 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 19567
W20260925 22:44:56.091384 19358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.74.193:46569 (error 108)
I20260925 22:44:56.095557 19841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:56.095237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:46569 (local address 127.0.0.1:46514)
0925 22:44:56.095391 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:56.095464 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.193:46569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 22:44:56.095818 19358 catalog_manager.cc:4584] TS a6c855a6dc3149d2a9f5e670a124b144 (127.2.74.193:46569): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d03ef7526fdc46f79ad560503c4d71d4: Network error: Client connection negotiation failed: client connection to 127.2.74.193:46569: connect: Connection refused (error 111)
I20260925 22:44:56.099879 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 19701
I20260925 22:44:56.105818 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 19339
2026-09-25T22:44:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2421 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T22:44:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:44:56Z Disabled control of system clock
I20260925 22:44:56.136874 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:39649
--webserver_interface=127.2.74.254
--webserver_port=0
--builtin_ntp_servers=127.2.74.212:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:39649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:44:56.257186 19877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:56.257498 19877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:56.257535 19877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:56.262022 19877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:44:56.262109 19877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:56.262125 19877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:56.262142 19877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:44:56.262158 19877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:44:56.267511 19877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:39649
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:39649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:44:56.268764 19877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:56.270283 19877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:56.279206 19882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:56.279196 19885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:56.279219 19883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:56.279448 19877 server_base.cc:1034] running on GCE node
I20260925 22:44:56.280087 19877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:56.281313 19877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:56.282498 19877 hybrid_clock.cc:648] HybridClock initialized: now 1790376296282465 us; error 51 us; skew 500 ppm
I20260925 22:44:56.284672 19877 webserver.cc:460] Webserver started at http://127.2.74.254:36589/ using document root <none> and password file <none>
I20260925 22:44:56.285331 19877 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:56.285416 19877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:56.285668 19877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:56.288425 19877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "0ffa20fe33954294bb926605ae187215"
format_stamp: "Formatted at 2026-09-25 22:44:56 on dist-test-slave-csnl"
server_key: "28529f26101615bc62022117b008ebd6"
server_key_iv: "83766d4f9e0eaecefd209062b6693247"
server_key_version: "encryptionkey@0"
I20260925 22:44:56.289075 19877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "0ffa20fe33954294bb926605ae187215"
format_stamp: "Formatted at 2026-09-25 22:44:56 on dist-test-slave-csnl"
server_key: "28529f26101615bc62022117b008ebd6"
server_key_iv: "83766d4f9e0eaecefd209062b6693247"
server_key_version: "encryptionkey@0"
I20260925 22:44:56.293679 19877 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:44:56.297117 19891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:56.298494 19877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:44:56.298655 19877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "0ffa20fe33954294bb926605ae187215"
format_stamp: "Formatted at 2026-09-25 22:44:56 on dist-test-slave-csnl"
server_key: "28529f26101615bc62022117b008ebd6"
server_key_iv: "83766d4f9e0eaecefd209062b6693247"
server_key_version: "encryptionkey@0"
I20260925 22:44:56.298802 19877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:56.313990 19877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:56.314690 19877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:56.314860 19877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:56.322463 19877 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:39649
I20260925 22:44:56.322471 19943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:39649 every 8 connection(s)
I20260925 22:44:56.323740 19877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:44:56.327337 19944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:56.333451 19944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ffa20fe33954294bb926605ae187215: Bootstrap starting.
I20260925 22:44:56.333896 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 19877
I20260925 22:44:56.334000 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:44:56.334221 2347 external_mini_cluster.cc:1428] Setting key 0278b50c3a3c3f9648280b3d9a22c1fc
I20260925 22:44:56.336385 19944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ffa20fe33954294bb926605ae187215: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:56.337227 19944 log.cc:824] T 00000000000000000000000000000000 P 0ffa20fe33954294bb926605ae187215: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:56.339669 19944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ffa20fe33954294bb926605ae187215: No bootstrap required, opened a new log
I20260925 22:44:56.344477 19944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ffa20fe33954294bb926605ae187215 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ffa20fe33954294bb926605ae187215" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 39649 } }
I20260925 22:44:56.344722 19947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:56.335684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41312 (local address 127.2.74.254:39649)
0925 22:44:56.336386 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:56.336401 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:56.336439 (+ 38us) server_negotiation.cc:407] Connection header received
0925 22:44:56.337197 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:56.337216 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:56.337790 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:56.338205 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:56.338597 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.339704 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.340983 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.341501 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.341638 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:56.342619 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:56.342625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:56.342640 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:44:56.343156 (+ 516us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:56.343221 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:44:56.343244 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:56.343334 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:56.343537 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 22:44:56.343867 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":156,"threads_started":1}
I20260925 22:44:56.344779 19944 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ffa20fe33954294bb926605ae187215 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:56.344846 19944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ffa20fe33954294bb926605ae187215 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ffa20fe33954294bb926605ae187215, State: Initialized, Role: FOLLOWER
I20260925 22:44:56.345343 19944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ffa20fe33954294bb926605ae187215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ffa20fe33954294bb926605ae187215" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 39649 } }
I20260925 22:44:56.345525 19944 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ffa20fe33954294bb926605ae187215 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:44:56.345638 19944 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ffa20fe33954294bb926605ae187215 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:44:56.345772 19944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ffa20fe33954294bb926605ae187215 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:56.347110 19944 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ffa20fe33954294bb926605ae187215 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ffa20fe33954294bb926605ae187215" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 39649 } }
I20260925 22:44:56.347512 19944 leader_election.cc:302] T 00000000000000000000000000000000 P 0ffa20fe33954294bb926605ae187215 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ffa20fe33954294bb926605ae187215; no voters:
I20260925 22:44:56.347859 19944 leader_election.cc:288] T 00000000000000000000000000000000 P 0ffa20fe33954294bb926605ae187215 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:44:56.348040 19949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ffa20fe33954294bb926605ae187215 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:56.348300 19949 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ffa20fe33954294bb926605ae187215 [term 1 LEADER]: Becoming Leader. State: Replica: 0ffa20fe33954294bb926605ae187215, State: Running, Role: LEADER
I20260925 22:44:56.348676 19949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ffa20fe33954294bb926605ae187215 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ffa20fe33954294bb926605ae187215" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 39649 } }
I20260925 22:44:56.349306 19944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ffa20fe33954294bb926605ae187215 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:44:56.350643 19951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ffa20fe33954294bb926605ae187215 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ffa20fe33954294bb926605ae187215. Latest consensus state: current_term: 1 leader_uuid: "0ffa20fe33954294bb926605ae187215" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ffa20fe33954294bb926605ae187215" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 39649 } } }
I20260925 22:44:56.350668 19950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ffa20fe33954294bb926605ae187215 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ffa20fe33954294bb926605ae187215" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ffa20fe33954294bb926605ae187215" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 39649 } } }
I20260925 22:44:56.350802 19951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ffa20fe33954294bb926605ae187215 [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:56.350802 19950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ffa20fe33954294bb926605ae187215 [sys.catalog]: This master's current role is: LEADER
W20260925 22:44:56.354485 19964 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ffa20fe33954294bb926605ae187215: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:44:56.354578 19964 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:44:56.354681 19958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:44:56.355654 19958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:44:56.361759 19958 catalog_manager.cc:1347] Generated new cluster ID: 3668a27d177a4558b19c03102ab53322
I20260925 22:44:56.361856 19958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:44:56.379055 19958 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:44:56.380458 19958 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:44:56.392647 19958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ffa20fe33954294bb926605ae187215: Generated new TSK 0
I20260925 22:44:56.393528 19958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:44:56.404663 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:0
--local_ip_for_outbound_sockets=127.2.74.193
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:39649
--builtin_ntp_servers=127.2.74.212:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:44:56.524609 19968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:56.524937 19968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:56.524979 19968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:56.529165 19968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:56.529235 19968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:56.529294 19968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:44:56.534487 19968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:39649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:44:56.535746 19968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:56.537195 19968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:56.548079 19973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:56.548074 19976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:56.548079 19974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:56.548386 19968 server_base.cc:1034] running on GCE node
I20260925 22:44:56.548990 19968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:56.549826 19968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:56.551030 19968 hybrid_clock.cc:648] HybridClock initialized: now 1790376296551007 us; error 46 us; skew 500 ppm
I20260925 22:44:56.553095 19968 webserver.cc:460] Webserver started at http://127.2.74.193:43619/ using document root <none> and password file <none>
I20260925 22:44:56.553762 19968 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:56.553869 19968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:56.554117 19968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:56.556443 19968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-0/data/instance:
uuid: "bdf4615c510f43868a52fd06e01e3ac7"
format_stamp: "Formatted at 2026-09-25 22:44:56 on dist-test-slave-csnl"
server_key: "a5f53bbf6610e709cccdd3e31fcd05e9"
server_key_iv: "ac4880d043aca3480b244177ecad6a23"
server_key_version: "encryptionkey@0"
I20260925 22:44:56.557065 19968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance:
uuid: "bdf4615c510f43868a52fd06e01e3ac7"
format_stamp: "Formatted at 2026-09-25 22:44:56 on dist-test-slave-csnl"
server_key: "a5f53bbf6610e709cccdd3e31fcd05e9"
server_key_iv: "ac4880d043aca3480b244177ecad6a23"
server_key_version: "encryptionkey@0"
I20260925 22:44:56.561228 19968 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:44:56.564077 19982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:56.565418 19968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:56.565598 19968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "bdf4615c510f43868a52fd06e01e3ac7"
format_stamp: "Formatted at 2026-09-25 22:44:56 on dist-test-slave-csnl"
server_key: "a5f53bbf6610e709cccdd3e31fcd05e9"
server_key_iv: "ac4880d043aca3480b244177ecad6a23"
server_key_version: "encryptionkey@0"
I20260925 22:44:56.565747 19968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:56.579244 19968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:56.580039 19968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:56.580265 19968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:56.580912 19968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:56.581981 19968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:56.582059 19968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:56.582136 19968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:56.582180 19968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:56.591513 19968 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:45153
I20260925 22:44:56.591538 20095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:45153 every 8 connection(s)
I20260925 22:44:56.592653 19968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:44:56.602175 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 19968
I20260925 22:44:56.602311 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance
I20260925 22:44:56.602638 2347 external_mini_cluster.cc:1428] Setting key 8fdf11954c3acd23e6e7f9c935e72fc3
I20260925 22:44:56.603724 19947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:56.594653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:59453 (local address 127.2.74.254:39649)
0925 22:44:56.594819 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:56.594823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:56.596135 (+ 1312us) server_negotiation.cc:407] Connection header received
0925 22:44:56.597710 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:56.597714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:56.597784 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:56.597901 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:56.598985 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.599529 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.601138 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.601377 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.601471 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:56.602982 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:56.602989 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:56.602996 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:44:56.603013 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:56.603039 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:44:56.603057 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:56.603153 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:56.603418 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 22:44:56.603525 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 22:44:56.604931 20098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:56.595189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:39649 (local address 127.2.74.193:59453)
0925 22:44:56.595942 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:56.595989 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:56.597428 (+ 1439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:56.598112 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:56.598120 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:56.598228 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:56.598779 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:56.598794 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.599682 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.599692 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:56.600958 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:56.600972 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.601569 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.601573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:56.601766 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:56.602574 (+ 808us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:44:56.602575 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:56.602627 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:44:56.602744 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:56.603200 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:56.603212 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:56.603219 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 22:44:56.603367 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 22:44:56.603778 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":133,"threads_started":1}
I20260925 22:44:56.605054 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:0
--local_ip_for_outbound_sockets=127.2.74.194
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:39649
--builtin_ntp_servers=127.2.74.212:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:44:56.606787 20096 heartbeater.cc:347] Connected to a master server at 127.2.74.254:39649
I20260925 22:44:56.607131 20096 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:56.607872 20096 heartbeater.cc:511] Master 127.2.74.254:39649 requested a full tablet report, sending...
I20260925 22:44:56.609836 19908 ts_manager.cc:194] Registered new tserver with Master: bdf4615c510f43868a52fd06e01e3ac7 (127.2.74.193:45153)
I20260925 22:44:56.611191 19908 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.74.193:59453
W20260925 22:44:56.722999 20099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:56.723364 20099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:56.723436 20099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:56.727715 20099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:56.727826 20099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:56.727926 20099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:44:56.733059 20099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:39649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:44:56.734217 20099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:56.735723 20099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:56.746572 20104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:56.746573 20107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:56.746573 20105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:56.746642 20099 server_base.cc:1034] running on GCE node
I20260925 22:44:56.747383 20099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:56.748083 20099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:56.749285 20099 hybrid_clock.cc:648] HybridClock initialized: now 1790376296749266 us; error 32 us; skew 500 ppm
I20260925 22:44:56.751516 20099 webserver.cc:460] Webserver started at http://127.2.74.194:37745/ using document root <none> and password file <none>
I20260925 22:44:56.752204 20099 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:56.752306 20099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:56.752542 20099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:56.754729 20099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-1/data/instance:
uuid: "b6d50b600d7646839c32e4436a71d042"
format_stamp: "Formatted at 2026-09-25 22:44:56 on dist-test-slave-csnl"
server_key: "10da595118f8b4c27c68e81f49681a7a"
server_key_iv: "6ef84776e981f88d64f3cc3ee195e2e5"
server_key_version: "encryptionkey@0"
I20260925 22:44:56.755404 20099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance:
uuid: "b6d50b600d7646839c32e4436a71d042"
format_stamp: "Formatted at 2026-09-25 22:44:56 on dist-test-slave-csnl"
server_key: "10da595118f8b4c27c68e81f49681a7a"
server_key_iv: "6ef84776e981f88d64f3cc3ee195e2e5"
server_key_version: "encryptionkey@0"
I20260925 22:44:56.759545 20099 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:44:56.762321 20113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:56.763792 20099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:56.763954 20099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "b6d50b600d7646839c32e4436a71d042"
format_stamp: "Formatted at 2026-09-25 22:44:56 on dist-test-slave-csnl"
server_key: "10da595118f8b4c27c68e81f49681a7a"
server_key_iv: "6ef84776e981f88d64f3cc3ee195e2e5"
server_key_version: "encryptionkey@0"
I20260925 22:44:56.764092 20099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:56.789294 20099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:56.790059 20099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:56.790277 20099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:56.790887 20099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:56.791916 20099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:56.791992 20099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:56.792064 20099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:56.792112 20099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:56.801779 20099 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:40991
I20260925 22:44:56.801920 20226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:40991 every 8 connection(s)
I20260925 22:44:56.803120 20099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:44:56.813024 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 20099
I20260925 22:44:56.813202 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance
I20260925 22:44:56.813293 19947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:56.805097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:44365 (local address 127.2.74.254:39649)
0925 22:44:56.805269 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:56.805273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:56.806448 (+ 1175us) server_negotiation.cc:407] Connection header received
0925 22:44:56.808024 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:56.808028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:56.808095 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:56.808182 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:56.809198 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.809766 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.811069 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.811309 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.811431 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:56.812733 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:56.812736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:56.812738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:44:56.812753 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:56.812775 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:44:56.812784 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:56.812868 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:56.813052 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 22:44:56.813129 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 22:44:56.813578 2347 external_mini_cluster.cc:1428] Setting key 3af0737b32d29ee85642c23563423050
I20260925 22:44:56.814445 20229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:56.805519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:39649 (local address 127.2.74.194:44365)
0925 22:44:56.806267 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:56.806310 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:56.807752 (+ 1442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:56.808392 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:56.808400 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:56.808498 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:56.809028 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:56.809041 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.809935 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.809938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:56.810887 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:56.810897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.811487 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.811490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:56.811676 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:56.812398 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:44:56.812400 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:56.812447 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:44:56.812525 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:56.812883 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:56.812896 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:56.812908 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 22:44:56.813047 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 22:44:56.813386 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":144,"threads_started":1}
I20260925 22:44:56.815747 20227 heartbeater.cc:347] Connected to a master server at 127.2.74.254:39649
I20260925 22:44:56.815913 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:0
--local_ip_for_outbound_sockets=127.2.74.195
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:39649
--builtin_ntp_servers=127.2.74.212:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:44:56.816036 20227 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:56.816702 20227 heartbeater.cc:511] Master 127.2.74.254:39649 requested a full tablet report, sending...
I20260925 22:44:56.817997 19908 ts_manager.cc:194] Registered new tserver with Master: b6d50b600d7646839c32e4436a71d042 (127.2.74.194:40991)
I20260925 22:44:56.818771 19908 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.74.194:44365
W20260925 22:44:56.941191 20230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:56.941507 20230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:56.941550 20230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:56.945919 20230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:56.945989 20230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:56.946049 20230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:44:56.951315 20230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:39649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:44:56.952576 20230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:56.954071 20230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:56.965081 20235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:56.965078 20238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:56.965131 20236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:56.965368 20230 server_base.cc:1034] running on GCE node
I20260925 22:44:56.965855 20230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:56.966562 20230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:56.967769 20230 hybrid_clock.cc:648] HybridClock initialized: now 1790376296967757 us; error 40 us; skew 500 ppm
I20260925 22:44:56.970007 20230 webserver.cc:460] Webserver started at http://127.2.74.195:42541/ using document root <none> and password file <none>
I20260925 22:44:56.970680 20230 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:56.970790 20230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:56.971088 20230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:56.973382 20230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-2/data/instance:
uuid: "001335f1ec5e4a27b8ac09294cddf7f5"
format_stamp: "Formatted at 2026-09-25 22:44:56 on dist-test-slave-csnl"
server_key: "8440b15c99d23cd0131a10e26772c72e"
server_key_iv: "8235cbe1d4c14ea57265f1e89bcb41ec"
server_key_version: "encryptionkey@0"
I20260925 22:44:56.974000 20230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance:
uuid: "001335f1ec5e4a27b8ac09294cddf7f5"
format_stamp: "Formatted at 2026-09-25 22:44:56 on dist-test-slave-csnl"
server_key: "8440b15c99d23cd0131a10e26772c72e"
server_key_iv: "8235cbe1d4c14ea57265f1e89bcb41ec"
server_key_version: "encryptionkey@0"
I20260925 22:44:56.978197 20230 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 22:44:56.981117 20244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:56.982540 20230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:44:56.982801 20230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "001335f1ec5e4a27b8ac09294cddf7f5"
format_stamp: "Formatted at 2026-09-25 22:44:56 on dist-test-slave-csnl"
server_key: "8440b15c99d23cd0131a10e26772c72e"
server_key_iv: "8235cbe1d4c14ea57265f1e89bcb41ec"
server_key_version: "encryptionkey@0"
I20260925 22:44:56.982997 20230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:57.017292 20230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:57.018216 20230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:57.018496 20230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:57.019291 20230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:57.020386 20230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:57.020468 20230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:57.020547 20230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:57.020599 20230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:57.031203 20230 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:43133
I20260925 22:44:57.031225 20357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:43133 every 8 connection(s)
I20260925 22:44:57.032248 20230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:44:57.033779 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 20230
I20260925 22:44:57.033881 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance
I20260925 22:44:57.034250 2347 external_mini_cluster.cc:1428] Setting key ae6a9b76b3f816fa39303ac84d58ed04
I20260925 22:44:57.042712 19947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:57.034349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:53391 (local address 127.2.74.254:39649)
0925 22:44:57.034503 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:57.034507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:57.035940 (+ 1433us) server_negotiation.cc:407] Connection header received
0925 22:44:57.037524 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:57.037527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:57.037588 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:57.037693 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:57.038703 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.039353 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.040617 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.040846 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.040972 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:57.042171 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:57.042180 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:57.042182 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:44:57.042196 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:57.042215 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:44:57.042224 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:57.042307 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:57.042482 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 22:44:57.042556 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:44:57.043735 20360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:57.034856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:39649 (local address 127.2.74.195:53391)
0925 22:44:57.035795 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:57.035844 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:57.037281 (+ 1437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:57.037889 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:57.037898 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:57.037995 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:57.038532 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:57.038547 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.039499 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.039502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:57.040459 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:57.040470 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.041025 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.041029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:57.041217 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:57.041859 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:44:57.041861 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:57.041910 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:44:57.041989 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:57.042308 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:57.042320 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:57.042328 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 22:44:57.042483 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 22:44:57.042809 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":156,"threads_started":1}
I20260925 22:44:57.044991 20358 heartbeater.cc:347] Connected to a master server at 127.2.74.254:39649
I20260925 22:44:57.045277 20358 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:57.045878 20358 heartbeater.cc:511] Master 127.2.74.254:39649 requested a full tablet report, sending...
I20260925 22:44:57.047173 19908 ts_manager.cc:194] Registered new tserver with Master: 001335f1ec5e4a27b8ac09294cddf7f5 (127.2.74.195:43133)
I20260925 22:44:57.047993 19908 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.74.195:53391
I20260925 22:44:57.048509 2347 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:44:57.054718 19947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:57.051048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41316 (local address 127.2.74.254:39649)
0925 22:44:57.051190 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:57.051195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:57.051425 (+ 230us) server_negotiation.cc:407] Connection header received
0925 22:44:57.051524 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:57.051527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:57.051570 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:57.051633 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:57.051867 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.052371 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.053206 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.053415 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.053493 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:57.054342 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:57.054346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:57.054348 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:44:57.054360 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:57.054378 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:44:57.054385 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:57.054444 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:57.054523 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 22:44:57.054576 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:44:57.057957 19908 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:44:57.061160 19908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:44:57.079706 20371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:57.071333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:40991 (local address 127.0.0.1:39166)
0925 22:44:57.072155 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:57.072171 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:57.072271 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:57.072829 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:57.072833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:57.072852 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:57.072976 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:57.072983 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.074665 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.074669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:57.076421 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:57.076436 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.078998 (+ 2562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.079002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:57.079149 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:57.079201 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 22:44:57.079313 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 22:44:57.079471 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"mutex_wait_us":32,"thread_start_us":127,"threads_started":1}
I20260925 22:44:57.080438 20370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:57.070895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32780 (local address 127.2.74.193:45153)
0925 22:44:57.071363 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:57.071372 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:57.071409 (+ 37us) server_negotiation.cc:407] Connection header received
0925 22:44:57.071740 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:57.071746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:57.071968 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:57.072126 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:57.073117 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.075102 (+ 1985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.077997 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.078921 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.079066 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:57.079304 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:57.080115 (+ 811us) server_negotiation.cc:299] Negotiation successful
0925 22:44:57.080252 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":110,"threads_started":1}
I20260925 22:44:57.080482 20373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:57.072162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:43133 (local address 127.0.0.1:48776)
0925 22:44:57.072729 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:57.072747 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:57.072844 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:57.073722 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:57.073726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:57.073747 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:57.073851 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:57.073863 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.076057 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.076065 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:57.077809 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:57.077821 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.080074 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.080081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:57.080176 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:57.080201 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:44:57.080268 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 22:44:57.080342 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":163,"thread_start_us":113,"threads_started":1}
I20260925 22:44:57.080884 20372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:57.071502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39166 (local address 127.2.74.194:40991)
0925 22:44:57.072302 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:57.072314 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:57.072336 (+ 22us) server_negotiation.cc:407] Connection header received
0925 22:44:57.072411 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:57.072416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:57.072604 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:57.073144 (+ 540us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:57.073281 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.074523 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.077736 (+ 3213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.078798 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.080220 (+ 1422us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:57.080468 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:57.080529 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 22:44:57.080665 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":121,"threads_started":1}
I20260925 22:44:57.081389 20374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:57.072612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48776 (local address 127.2.74.195:43133)
0925 22:44:57.073247 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:57.073260 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:57.073314 (+ 54us) server_negotiation.cc:407] Connection header received
0925 22:44:57.073400 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:57.073406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:57.073568 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:57.073740 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:57.074003 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.075904 (+ 1901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.077981 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.079929 (+ 1948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.080778 (+ 849us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:57.080996 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:57.081052 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 22:44:57.081160 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":150,"threads_started":1}
I20260925 22:44:57.081944 20369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:57.070813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:45153 (local address 127.0.0.1:32780)
0925 22:44:57.071246 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:57.071332 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:57.071482 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:57.072340 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:57.072352 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:57.072379 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:57.072977 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:57.072983 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.076092 (+ 3109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.076095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:57.077838 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:57.077860 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.079375 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.079379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:57.079479 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:57.079505 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 22:44:57.079581 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 22:44:57.079660 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":302,"thread_start_us":947,"threads_started":1}
I20260925 22:44:57.084476 20161 tablet_service.cc:1467] Processing CreateTablet for tablet 37dfdaed02764913919df9b26257b520 (DEFAULT_TABLE table=test-table [id=9d3a9bf241c64da1b396180aee6f4ea6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:57.084756 20030 tablet_service.cc:1467] Processing CreateTablet for tablet 37dfdaed02764913919df9b26257b520 (DEFAULT_TABLE table=test-table [id=9d3a9bf241c64da1b396180aee6f4ea6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:57.084714 20292 tablet_service.cc:1467] Processing CreateTablet for tablet 37dfdaed02764913919df9b26257b520 (DEFAULT_TABLE table=test-table [id=9d3a9bf241c64da1b396180aee6f4ea6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:57.085775 20161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37dfdaed02764913919df9b26257b520. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:57.085775 20030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37dfdaed02764913919df9b26257b520. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:57.085872 20292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37dfdaed02764913919df9b26257b520. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:57.094249 20375 tablet_bootstrap.cc:492] T 37dfdaed02764913919df9b26257b520 P 001335f1ec5e4a27b8ac09294cddf7f5: Bootstrap starting.
I20260925 22:44:57.096091 20377 tablet_bootstrap.cc:492] T 37dfdaed02764913919df9b26257b520 P b6d50b600d7646839c32e4436a71d042: Bootstrap starting.
I20260925 22:44:57.096939 20375 tablet_bootstrap.cc:654] T 37dfdaed02764913919df9b26257b520 P 001335f1ec5e4a27b8ac09294cddf7f5: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:57.097720 20375 log.cc:824] T 37dfdaed02764913919df9b26257b520 P 001335f1ec5e4a27b8ac09294cddf7f5: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:57.098539 20376 tablet_bootstrap.cc:492] T 37dfdaed02764913919df9b26257b520 P bdf4615c510f43868a52fd06e01e3ac7: Bootstrap starting.
I20260925 22:44:57.098673 20377 tablet_bootstrap.cc:654] T 37dfdaed02764913919df9b26257b520 P b6d50b600d7646839c32e4436a71d042: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:57.099602 20377 log.cc:824] T 37dfdaed02764913919df9b26257b520 P b6d50b600d7646839c32e4436a71d042: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:57.100699 20375 tablet_bootstrap.cc:492] T 37dfdaed02764913919df9b26257b520 P 001335f1ec5e4a27b8ac09294cddf7f5: No bootstrap required, opened a new log
I20260925 22:44:57.100875 20376 tablet_bootstrap.cc:654] T 37dfdaed02764913919df9b26257b520 P bdf4615c510f43868a52fd06e01e3ac7: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:57.101037 20375 ts_tablet_manager.cc:1397] T 37dfdaed02764913919df9b26257b520 P 001335f1ec5e4a27b8ac09294cddf7f5: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:44:57.101562 20377 tablet_bootstrap.cc:492] T 37dfdaed02764913919df9b26257b520 P b6d50b600d7646839c32e4436a71d042: No bootstrap required, opened a new log
I20260925 22:44:57.101785 20377 ts_tablet_manager.cc:1397] T 37dfdaed02764913919df9b26257b520 P b6d50b600d7646839c32e4436a71d042: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:44:57.101778 20376 log.cc:824] T 37dfdaed02764913919df9b26257b520 P bdf4615c510f43868a52fd06e01e3ac7: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:57.104055 20376 tablet_bootstrap.cc:492] T 37dfdaed02764913919df9b26257b520 P bdf4615c510f43868a52fd06e01e3ac7: No bootstrap required, opened a new log
I20260925 22:44:57.104251 20376 ts_tablet_manager.cc:1397] T 37dfdaed02764913919df9b26257b520 P bdf4615c510f43868a52fd06e01e3ac7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:44:57.105849 20375 raft_consensus.cc:348] T 37dfdaed02764913919df9b26257b520 P 001335f1ec5e4a27b8ac09294cddf7f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf4615c510f43868a52fd06e01e3ac7" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 45153 } } peers { permanent_uuid: "001335f1ec5e4a27b8ac09294cddf7f5" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43133 } } peers { permanent_uuid: "b6d50b600d7646839c32e4436a71d042" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 40991 } }
I20260925 22:44:57.105854 20377 raft_consensus.cc:348] T 37dfdaed02764913919df9b26257b520 P b6d50b600d7646839c32e4436a71d042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf4615c510f43868a52fd06e01e3ac7" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 45153 } } peers { permanent_uuid: "001335f1ec5e4a27b8ac09294cddf7f5" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43133 } } peers { permanent_uuid: "b6d50b600d7646839c32e4436a71d042" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 40991 } }
I20260925 22:44:57.106141 20375 raft_consensus.cc:374] T 37dfdaed02764913919df9b26257b520 P 001335f1ec5e4a27b8ac09294cddf7f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:57.106145 20377 raft_consensus.cc:374] T 37dfdaed02764913919df9b26257b520 P b6d50b600d7646839c32e4436a71d042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:57.106194 20375 raft_consensus.cc:729] T 37dfdaed02764913919df9b26257b520 P 001335f1ec5e4a27b8ac09294cddf7f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001335f1ec5e4a27b8ac09294cddf7f5, State: Initialized, Role: FOLLOWER
I20260925 22:44:57.106204 20377 raft_consensus.cc:729] T 37dfdaed02764913919df9b26257b520 P b6d50b600d7646839c32e4436a71d042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6d50b600d7646839c32e4436a71d042, State: Initialized, Role: FOLLOWER
I20260925 22:44:57.106674 20377 consensus_queue.cc:260] T 37dfdaed02764913919df9b26257b520 P b6d50b600d7646839c32e4436a71d042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf4615c510f43868a52fd06e01e3ac7" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 45153 } } peers { permanent_uuid: "001335f1ec5e4a27b8ac09294cddf7f5" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43133 } } peers { permanent_uuid: "b6d50b600d7646839c32e4436a71d042" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 40991 } }
I20260925 22:44:57.106675 20375 consensus_queue.cc:260] T 37dfdaed02764913919df9b26257b520 P 001335f1ec5e4a27b8ac09294cddf7f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf4615c510f43868a52fd06e01e3ac7" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 45153 } } peers { permanent_uuid: "001335f1ec5e4a27b8ac09294cddf7f5" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43133 } } peers { permanent_uuid: "b6d50b600d7646839c32e4436a71d042" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 40991 } }
I20260925 22:44:57.107484 20358 heartbeater.cc:503] Master 127.2.74.254:39649 was elected leader, sending a full tablet report...
I20260925 22:44:57.107832 20375 ts_tablet_manager.cc:1428] T 37dfdaed02764913919df9b26257b520 P 001335f1ec5e4a27b8ac09294cddf7f5: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:44:57.107837 20377 ts_tablet_manager.cc:1428] T 37dfdaed02764913919df9b26257b520 P b6d50b600d7646839c32e4436a71d042: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260925 22:44:57.108172 20227 heartbeater.cc:503] Master 127.2.74.254:39649 was elected leader, sending a full tablet report...
I20260925 22:44:57.109766 20376 raft_consensus.cc:348] T 37dfdaed02764913919df9b26257b520 P bdf4615c510f43868a52fd06e01e3ac7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf4615c510f43868a52fd06e01e3ac7" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 45153 } } peers { permanent_uuid: "001335f1ec5e4a27b8ac09294cddf7f5" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43133 } } peers { permanent_uuid: "b6d50b600d7646839c32e4436a71d042" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 40991 } }
I20260925 22:44:57.110045 20376 raft_consensus.cc:374] T 37dfdaed02764913919df9b26257b520 P bdf4615c510f43868a52fd06e01e3ac7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:57.110105 20376 raft_consensus.cc:729] T 37dfdaed02764913919df9b26257b520 P bdf4615c510f43868a52fd06e01e3ac7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf4615c510f43868a52fd06e01e3ac7, State: Initialized, Role: FOLLOWER
I20260925 22:44:57.110563 20376 consensus_queue.cc:260] T 37dfdaed02764913919df9b26257b520 P bdf4615c510f43868a52fd06e01e3ac7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf4615c510f43868a52fd06e01e3ac7" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 45153 } } peers { permanent_uuid: "001335f1ec5e4a27b8ac09294cddf7f5" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43133 } } peers { permanent_uuid: "b6d50b600d7646839c32e4436a71d042" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 40991 } }
I20260925 22:44:57.111275 20096 heartbeater.cc:503] Master 127.2.74.254:39649 was elected leader, sending a full tablet report...
I20260925 22:44:57.111737 20376 ts_tablet_manager.cc:1428] T 37dfdaed02764913919df9b26257b520 P bdf4615c510f43868a52fd06e01e3ac7: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 22:44:57.221164 20383 raft_consensus.cc:482] T 37dfdaed02764913919df9b26257b520 P bdf4615c510f43868a52fd06e01e3ac7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:44:57.221370 20383 raft_consensus.cc:504] T 37dfdaed02764913919df9b26257b520 P bdf4615c510f43868a52fd06e01e3ac7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf4615c510f43868a52fd06e01e3ac7" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 45153 } } peers { permanent_uuid: "001335f1ec5e4a27b8ac09294cddf7f5" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43133 } } peers { permanent_uuid: "b6d50b600d7646839c32e4436a71d042" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 40991 } }
I20260925 22:44:57.222489 20383 leader_election.cc:288] T 37dfdaed02764913919df9b26257b520 P bdf4615c510f43868a52fd06e01e3ac7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 001335f1ec5e4a27b8ac09294cddf7f5 (127.2.74.195:43133), b6d50b600d7646839c32e4436a71d042 (127.2.74.194:40991)
I20260925 22:44:57.226861 20374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:57.222928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:34285 (local address 127.2.74.195:43133)
0925 22:44:57.223066 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:57.223071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:57.223367 (+ 296us) server_negotiation.cc:407] Connection header received
0925 22:44:57.223525 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:57.223528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:57.223578 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:57.223666 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:57.224007 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.224569 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.225785 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.226215 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.226349 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:57.226402 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:57.226637 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 22:44:57.226712 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:44:57.226909 20385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:57.222800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:43133 (local address 127.2.74.193:34285)
0925 22:44:57.223258 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:57.223270 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:57.223403 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:57.223677 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:57.223680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:57.223726 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:57.223889 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:57.223894 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.224700 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.224704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:57.225632 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:57.225643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.226348 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.226352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:57.226504 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:57.226524 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 22:44:57.226618 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 22:44:57.226706 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":119,"thread_start_us":167,"threads_started":1}
I20260925 22:44:57.226922 20372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:57.222918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:37701 (local address 127.2.74.194:40991)
0925 22:44:57.223112 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:57.223116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:57.223302 (+ 186us) server_negotiation.cc:407] Connection header received
0925 22:44:57.223485 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:57.223488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:57.223546 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:57.223640 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:57.223999 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.224569 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.225833 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.226269 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.226419 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:57.226477 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:57.226703 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 22:44:57.226780 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260925 22:44:57.227216 20384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:57.222786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:40991 (local address 127.2.74.193:37701)
0925 22:44:57.223185 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:57.223199 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:57.223356 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:57.223670 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:57.223673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:57.223692 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:57.223872 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:57.223879 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.224703 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.224710 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:57.225689 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:57.225699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.226419 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.226424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:57.226562 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:57.226581 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 22:44:57.226682 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 22:44:57.226763 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":1,"spinlock_wait_cycles":3072,"thread_start_us":282,"threads_started":1}
I20260925 22:44:57.228051 20312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37dfdaed02764913919df9b26257b520" candidate_uuid: "bdf4615c510f43868a52fd06e01e3ac7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "001335f1ec5e4a27b8ac09294cddf7f5" is_pre_election: true
I20260925 22:44:57.228108 20181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37dfdaed02764913919df9b26257b520" candidate_uuid: "bdf4615c510f43868a52fd06e01e3ac7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6d50b600d7646839c32e4436a71d042" is_pre_election: true
I20260925 22:44:57.228458 20312 raft_consensus.cc:2454] T 37dfdaed02764913919df9b26257b520 P 001335f1ec5e4a27b8ac09294cddf7f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdf4615c510f43868a52fd06e01e3ac7 in term 0.
I20260925 22:44:57.228469 20181 raft_consensus.cc:2454] T 37dfdaed02764913919df9b26257b520 P b6d50b600d7646839c32e4436a71d042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdf4615c510f43868a52fd06e01e3ac7 in term 0.
I20260925 22:44:57.229030 19986 leader_election.cc:302] T 37dfdaed02764913919df9b26257b520 P bdf4615c510f43868a52fd06e01e3ac7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 001335f1ec5e4a27b8ac09294cddf7f5, b6d50b600d7646839c32e4436a71d042, bdf4615c510f43868a52fd06e01e3ac7; no voters:
I20260925 22:44:57.229331 20383 raft_consensus.cc:2781] T 37dfdaed02764913919df9b26257b520 P bdf4615c510f43868a52fd06e01e3ac7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:44:57.229421 20383 raft_consensus.cc:482] T 37dfdaed02764913919df9b26257b520 P bdf4615c510f43868a52fd06e01e3ac7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:44:57.229463 20383 raft_consensus.cc:3037] T 37dfdaed02764913919df9b26257b520 P bdf4615c510f43868a52fd06e01e3ac7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:57.230510 20383 raft_consensus.cc:504] T 37dfdaed02764913919df9b26257b520 P bdf4615c510f43868a52fd06e01e3ac7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf4615c510f43868a52fd06e01e3ac7" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 45153 } } peers { permanent_uuid: "001335f1ec5e4a27b8ac09294cddf7f5" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43133 } } peers { permanent_uuid: "b6d50b600d7646839c32e4436a71d042" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 40991 } }
I20260925 22:44:57.230898 20383 leader_election.cc:288] T 37dfdaed02764913919df9b26257b520 P bdf4615c510f43868a52fd06e01e3ac7 [CANDIDATE]: Term 1 election: Requested vote from peers 001335f1ec5e4a27b8ac09294cddf7f5 (127.2.74.195:43133), b6d50b600d7646839c32e4436a71d042 (127.2.74.194:40991)
I20260925 22:44:57.231300 20312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37dfdaed02764913919df9b26257b520" candidate_uuid: "bdf4615c510f43868a52fd06e01e3ac7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "001335f1ec5e4a27b8ac09294cddf7f5"
I20260925 22:44:57.231357 20181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37dfdaed02764913919df9b26257b520" candidate_uuid: "bdf4615c510f43868a52fd06e01e3ac7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6d50b600d7646839c32e4436a71d042"
I20260925 22:44:57.231448 20312 raft_consensus.cc:3037] T 37dfdaed02764913919df9b26257b520 P 001335f1ec5e4a27b8ac09294cddf7f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:57.231465 20181 raft_consensus.cc:3037] T 37dfdaed02764913919df9b26257b520 P b6d50b600d7646839c32e4436a71d042 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:57.232815 20312 raft_consensus.cc:2454] T 37dfdaed02764913919df9b26257b520 P 001335f1ec5e4a27b8ac09294cddf7f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdf4615c510f43868a52fd06e01e3ac7 in term 1.
I20260925 22:44:57.232818 20181 raft_consensus.cc:2454] T 37dfdaed02764913919df9b26257b520 P b6d50b600d7646839c32e4436a71d042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdf4615c510f43868a52fd06e01e3ac7 in term 1.
I20260925 22:44:57.233206 19983 leader_election.cc:302] T 37dfdaed02764913919df9b26257b520 P bdf4615c510f43868a52fd06e01e3ac7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6d50b600d7646839c32e4436a71d042, bdf4615c510f43868a52fd06e01e3ac7; no voters:
I20260925 22:44:57.233419 20383 raft_consensus.cc:2781] T 37dfdaed02764913919df9b26257b520 P bdf4615c510f43868a52fd06e01e3ac7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:57.233657 20383 raft_consensus.cc:686] T 37dfdaed02764913919df9b26257b520 P bdf4615c510f43868a52fd06e01e3ac7 [term 1 LEADER]: Becoming Leader. State: Replica: bdf4615c510f43868a52fd06e01e3ac7, State: Running, Role: LEADER
I20260925 22:44:57.234076 20383 consensus_queue.cc:237] T 37dfdaed02764913919df9b26257b520 P bdf4615c510f43868a52fd06e01e3ac7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf4615c510f43868a52fd06e01e3ac7" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 45153 } } peers { permanent_uuid: "001335f1ec5e4a27b8ac09294cddf7f5" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43133 } } peers { permanent_uuid: "b6d50b600d7646839c32e4436a71d042" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 40991 } }
I20260925 22:44:57.237582 19907 catalog_manager.cc:5495] T 37dfdaed02764913919df9b26257b520 P bdf4615c510f43868a52fd06e01e3ac7 reported cstate change: term changed from 0 to 1, leader changed from <none> to bdf4615c510f43868a52fd06e01e3ac7 (127.2.74.193). New cstate: current_term: 1 leader_uuid: "bdf4615c510f43868a52fd06e01e3ac7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf4615c510f43868a52fd06e01e3ac7" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 45153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001335f1ec5e4a27b8ac09294cddf7f5" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6d50b600d7646839c32e4436a71d042" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 40991 } health_report { overall_health: UNKNOWN } } }
I20260925 22:44:57.274678 20370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:57.270874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32784 (local address 127.2.74.193:45153)
0925 22:44:57.271010 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:57.271017 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:57.271076 (+ 59us) server_negotiation.cc:407] Connection header received
0925 22:44:57.271296 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:57.271298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:57.271361 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:57.271477 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:44:57.271715 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.272247 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.273121 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.273349 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.273465 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:57.273803 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:44:57.274252 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:44:57.274395 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:57.274456 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 22:44:57.274518 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 22:44:57.281205 20312 raft_consensus.cc:1261] T 37dfdaed02764913919df9b26257b520 P 001335f1ec5e4a27b8ac09294cddf7f5 [term 1 FOLLOWER]: Refusing update from remote peer bdf4615c510f43868a52fd06e01e3ac7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:57.281206 20181 raft_consensus.cc:1261] T 37dfdaed02764913919df9b26257b520 P b6d50b600d7646839c32e4436a71d042 [term 1 FOLLOWER]: Refusing update from remote peer bdf4615c510f43868a52fd06e01e3ac7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:57.282047 20386 consensus_queue.cc:1035] T 37dfdaed02764913919df9b26257b520 P bdf4615c510f43868a52fd06e01e3ac7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6d50b600d7646839c32e4436a71d042" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 40991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:57.282284 20383 consensus_queue.cc:1035] T 37dfdaed02764913919df9b26257b520 P bdf4615c510f43868a52fd06e01e3ac7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "001335f1ec5e4a27b8ac09294cddf7f5" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 22:44:57.283282 20359 tablet.cc:2369] T 37dfdaed02764913919df9b26257b520 P 001335f1ec5e4a27b8ac09294cddf7f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:44:57.289489 20389 mvcc.cc:205] Tried to move back new op lower bound from 7333381313655865344 to 7333381313473871872. Current Snapshot: MvccSnapshot[applied={T|T < 7333381313655865344}]
I20260925 22:44:57.290685 20391 mvcc.cc:205] Tried to move back new op lower bound from 7333381313655865344 to 7333381313473871872. Current Snapshot: MvccSnapshot[applied={T|T < 7333381313655865344}]
I20260925 22:44:57.291249 20390 mvcc.cc:205] Tried to move back new op lower bound from 7333381313655865344 to 7333381313473871872. Current Snapshot: MvccSnapshot[applied={T|T < 7333381313655865344}]
I20260925 22:44:57.300952 19908 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:57.301219 19908 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:57.304148 19908 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ffa20fe33954294bb926605ae187215: Sending DeleteTablet for 3 replicas of tablet 37dfdaed02764913919df9b26257b520
I20260925 22:44:57.305179 20030 tablet_service.cc:1514] Processing DeleteTablet for tablet 37dfdaed02764913919df9b26257b520 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:57 UTC) from {username='slave'} at 127.0.0.1:32780
I20260925 22:44:57.305599 20161 tablet_service.cc:1514] Processing DeleteTablet for tablet 37dfdaed02764913919df9b26257b520 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:57 UTC) from {username='slave'} at 127.0.0.1:39166
I20260925 22:44:57.305784 20399 tablet_replica.cc:330] T 37dfdaed02764913919df9b26257b520 P bdf4615c510f43868a52fd06e01e3ac7: stopping tablet replica
I20260925 22:44:57.306090 20400 tablet_replica.cc:330] T 37dfdaed02764913919df9b26257b520 P b6d50b600d7646839c32e4436a71d042: stopping tablet replica
I20260925 22:44:57.306126 20292 tablet_service.cc:1514] Processing DeleteTablet for tablet 37dfdaed02764913919df9b26257b520 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:57 UTC) from {username='slave'} at 127.0.0.1:48776
I20260925 22:44:57.306208 20399 raft_consensus.cc:2229] T 37dfdaed02764913919df9b26257b520 P bdf4615c510f43868a52fd06e01e3ac7 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:44:57.306371 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 19968
I20260925 22:44:57.306563 20400 raft_consensus.cc:2229] T 37dfdaed02764913919df9b26257b520 P b6d50b600d7646839c32e4436a71d042 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:57.306605 20399 raft_consensus.cc:2258] T 37dfdaed02764913919df9b26257b520 P bdf4615c510f43868a52fd06e01e3ac7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:57.307107 20400 raft_consensus.cc:2258] T 37dfdaed02764913919df9b26257b520 P b6d50b600d7646839c32e4436a71d042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:57.308604 20401 tablet_replica.cc:330] T 37dfdaed02764913919df9b26257b520 P 001335f1ec5e4a27b8ac09294cddf7f5: stopping tablet replica
I20260925 22:44:57.308668 20400 ts_tablet_manager.cc:1905] T 37dfdaed02764913919df9b26257b520 P b6d50b600d7646839c32e4436a71d042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:57.308979 20401 raft_consensus.cc:2229] T 37dfdaed02764913919df9b26257b520 P 001335f1ec5e4a27b8ac09294cddf7f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:57.309497 20401 raft_consensus.cc:2258] T 37dfdaed02764913919df9b26257b520 P 001335f1ec5e4a27b8ac09294cddf7f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:57.311256 20401 ts_tablet_manager.cc:1905] T 37dfdaed02764913919df9b26257b520 P 001335f1ec5e4a27b8ac09294cddf7f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:57.311494 20400 ts_tablet_manager.cc:1918] T 37dfdaed02764913919df9b26257b520 P b6d50b600d7646839c32e4436a71d042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:44:57.311614 20400 log.cc:1196] T 37dfdaed02764913919df9b26257b520 P b6d50b600d7646839c32e4436a71d042: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-1/wal/wals/37dfdaed02764913919df9b26257b520
I20260925 22:44:57.311932 20400 ts_tablet_manager.cc:1939] T 37dfdaed02764913919df9b26257b520 P b6d50b600d7646839c32e4436a71d042: Deleting consensus metadata
I20260925 22:44:57.313475 19892 catalog_manager.cc:4841] TS b6d50b600d7646839c32e4436a71d042 (127.2.74.194:40991): tablet 37dfdaed02764913919df9b26257b520 (table test-table [id=9d3a9bf241c64da1b396180aee6f4ea6]) successfully deleted
I20260925 22:44:57.314823 20401 ts_tablet_manager.cc:1918] T 37dfdaed02764913919df9b26257b520 P 001335f1ec5e4a27b8ac09294cddf7f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:44:57.314972 20401 log.cc:1196] T 37dfdaed02764913919df9b26257b520 P 001335f1ec5e4a27b8ac09294cddf7f5: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376047580784-2347-0/minicluster-data/ts-2/wal/wals/37dfdaed02764913919df9b26257b520
I20260925 22:44:57.315315 20401 ts_tablet_manager.cc:1939] T 37dfdaed02764913919df9b26257b520 P 001335f1ec5e4a27b8ac09294cddf7f5: Deleting consensus metadata
W20260925 22:44:57.315845 19895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.74.193:45153 (error 108)
I20260925 22:44:57.316744 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 20099
I20260925 22:44:57.316869 19895 catalog_manager.cc:4841] TS 001335f1ec5e4a27b8ac09294cddf7f5 (127.2.74.195:43133): tablet 37dfdaed02764913919df9b26257b520 (table test-table [id=9d3a9bf241c64da1b396180aee6f4ea6]) successfully deleted
I20260925 22:44:57.316998 20369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:57.316685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:45153 (local address 127.0.0.1:42224)
0925 22:44:57.316827 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:57.316910 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.193:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 22:44:57.317375 19895 catalog_manager.cc:4584] TS bdf4615c510f43868a52fd06e01e3ac7 (127.2.74.193:45153): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 37dfdaed02764913919df9b26257b520: Network error: Client connection negotiation failed: client connection to 127.2.74.193:45153: connect: Connection refused (error 111)
I20260925 22:44:57.323401 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 20230
I20260925 22:44:57.331398 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 19877
2026-09-25T22:44:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1223 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:44:57 dist-test-slave-csnl krb5kdc[20405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:44:57 dist-test-slave-csnl krb5kdc[20405](info): set up 2 sockets
Sep 25 22:44:57 dist-test-slave-csnl krb5kdc[20405](info): commencing operation
krb5kdc: starting...
W20260925 22:44:58.375066 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:44:58 dist-test-slave-csnl krb5kdc[20405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:44:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:44:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:44:58.502909 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:33599
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:44061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:33599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:58.620139 20418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:58.620448 20418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:58.620492 20418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:58.624656 20418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:44:58.624730 20418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:58.624750 20418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:58.624768 20418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:44:58.624786 20418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:44:58.630802 20418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:44061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:33599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:33599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:44:58.631946 20418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:58.632941 20418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:44:58.641894 20418 server_base.cc:1034] running on GCE node
W20260925 22:44:58.641808 20424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:58.641853 20426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:58.641804 20423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:58.642689 20418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:58.643896 20418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:58.645100 20418 hybrid_clock.cc:648] HybridClock initialized: now 1790376298645069 us; error 49 us; skew 500 ppm
Sep 25 22:44:58 dist-test-slave-csnl krb5kdc[20405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376298, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:58.648020 20418 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:44:58.649025 20418 webserver.cc:460] Webserver started at http://127.2.74.254:38711/ using document root <none> and password file <none>
I20260925 22:44:58.649618 20418 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:58.649711 20418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:58.649967 20418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:58.652181 20418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "7427acd1a97d462f843784ff57bb1eb2"
format_stamp: "Formatted at 2026-09-25 22:44:58 on dist-test-slave-csnl"
server_key: "9a0bcc0f5ef729a4b1abf2374bbbbf6c"
server_key_iv: "40cee84d72ce07a4c8f9df22f98cfe42"
server_key_version: "encryptionkey@0"
I20260925 22:44:58.652789 20418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "7427acd1a97d462f843784ff57bb1eb2"
format_stamp: "Formatted at 2026-09-25 22:44:58 on dist-test-slave-csnl"
server_key: "9a0bcc0f5ef729a4b1abf2374bbbbf6c"
server_key_iv: "40cee84d72ce07a4c8f9df22f98cfe42"
server_key_version: "encryptionkey@0"
I20260925 22:44:58.656762 20418 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:44:58.659492 20433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:58.660903 20418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:58.661084 20418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "7427acd1a97d462f843784ff57bb1eb2"
format_stamp: "Formatted at 2026-09-25 22:44:58 on dist-test-slave-csnl"
server_key: "9a0bcc0f5ef729a4b1abf2374bbbbf6c"
server_key_iv: "40cee84d72ce07a4c8f9df22f98cfe42"
server_key_version: "encryptionkey@0"
I20260925 22:44:58.661226 20418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:58.688920 20418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:58.692395 20418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:58.692641 20418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:58.700820 20418 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:33599
I20260925 22:44:58.700834 20485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:33599 every 8 connection(s)
I20260925 22:44:58.701988 20418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:44:58.705202 20486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:58.710242 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 20418
I20260925 22:44:58.710415 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:44:58.710747 2347 external_mini_cluster.cc:1428] Setting key b021e62574dd038e9b81d81d61919546
I20260925 22:44:58.712018 20486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7427acd1a97d462f843784ff57bb1eb2: Bootstrap starting.
I20260925 22:44:58.714632 20486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7427acd1a97d462f843784ff57bb1eb2: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:58.715605 20486 log.cc:824] T 00000000000000000000000000000000 P 7427acd1a97d462f843784ff57bb1eb2: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:58.717737 20486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7427acd1a97d462f843784ff57bb1eb2: No bootstrap required, opened a new log
Sep 25 22:44:58 dist-test-slave-csnl krb5kdc[20405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:44:58.722483 20486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7427acd1a97d462f843784ff57bb1eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7427acd1a97d462f843784ff57bb1eb2" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33599 } }
I20260925 22:44:58.722762 20486 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7427acd1a97d462f843784ff57bb1eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:58.722836 20486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7427acd1a97d462f843784ff57bb1eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7427acd1a97d462f843784ff57bb1eb2, State: Initialized, Role: FOLLOWER
I20260925 22:44:58.723342 20486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7427acd1a97d462f843784ff57bb1eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7427acd1a97d462f843784ff57bb1eb2" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33599 } }
I20260925 22:44:58.723510 20486 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7427acd1a97d462f843784ff57bb1eb2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:44:58.723589 20486 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7427acd1a97d462f843784ff57bb1eb2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:44:58.723742 20486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7427acd1a97d462f843784ff57bb1eb2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:58.724893 20486 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7427acd1a97d462f843784ff57bb1eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7427acd1a97d462f843784ff57bb1eb2" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33599 } }
I20260925 22:44:58.725320 20486 leader_election.cc:302] T 00000000000000000000000000000000 P 7427acd1a97d462f843784ff57bb1eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7427acd1a97d462f843784ff57bb1eb2; no voters:
I20260925 22:44:58.725737 20486 leader_election.cc:288] T 00000000000000000000000000000000 P 7427acd1a97d462f843784ff57bb1eb2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:44:58.726038 20491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7427acd1a97d462f843784ff57bb1eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:58.726293 20491 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7427acd1a97d462f843784ff57bb1eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 7427acd1a97d462f843784ff57bb1eb2, State: Running, Role: LEADER
I20260925 22:44:58.726750 20491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7427acd1a97d462f843784ff57bb1eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7427acd1a97d462f843784ff57bb1eb2" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33599 } }
I20260925 22:44:58.727142 20489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:58.712407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41120 (local address 127.2.74.254:33599)
0925 22:44:58.712906 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:58.712922 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:58.712964 (+ 42us) server_negotiation.cc:407] Connection header received
0925 22:44:58.713707 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:58.713728 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:58.714124 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:58.714526 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:58.715499 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.716585 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.717514 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.718110 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.718320 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:58.721267 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:58.721273 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:58.721293 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:58.721328 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:58.723357 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:58.723971 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:58.723979 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:58.723989 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:58.724167 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:58.724471 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:58.724474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:58.724476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:58.724939 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:58.725162 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:58.725625 (+ 463us) server_negotiation.cc:299] Negotiation successful
0925 22:44:58.725994 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":177,"threads_started":1}
I20260925 22:44:58.727175 20486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7427acd1a97d462f843784ff57bb1eb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:44:58.728545 20493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7427acd1a97d462f843784ff57bb1eb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7427acd1a97d462f843784ff57bb1eb2. Latest consensus state: current_term: 1 leader_uuid: "7427acd1a97d462f843784ff57bb1eb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7427acd1a97d462f843784ff57bb1eb2" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33599 } } }
I20260925 22:44:58.728714 20493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7427acd1a97d462f843784ff57bb1eb2 [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:58.729074 20492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7427acd1a97d462f843784ff57bb1eb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7427acd1a97d462f843784ff57bb1eb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7427acd1a97d462f843784ff57bb1eb2" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33599 } } }
I20260925 22:44:58.729175 20492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7427acd1a97d462f843784ff57bb1eb2 [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:58.729632 20502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:44:58.732682 20502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:44:58.738862 20502 catalog_manager.cc:1347] Generated new cluster ID: 9abe6e8d7c0f4b8589fb8047f79ea074
I20260925 22:44:58.738981 20502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:44:58.750672 20502 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:44:58.752032 20502 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:44:58.758246 20502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7427acd1a97d462f843784ff57bb1eb2: Generated new TSK 0
I20260925 22:44:58.759151 20502 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
I20260925 22:44:58.787572 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:0
--local_ip_for_outbound_sockets=127.2.74.193
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:33599
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:44061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:58.909158 20512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:58.909521 20512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:58.909605 20512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:58.914062 20512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:58.914182 20512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:58.914284 20512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:44:58.920053 20512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:44061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:33599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:44:58.921270 20512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:58.922369 20512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:58.932554 20517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:58.932538 20520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:58.932677 20512 server_base.cc:1034] running on GCE node
W20260925 22:44:58.932538 20518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:58.933264 20512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:58.933950 20512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:58.935161 20512 hybrid_clock.cc:648] HybridClock initialized: now 1790376298935144 us; error 32 us; skew 500 ppm
Sep 25 22:44:58 dist-test-slave-csnl krb5kdc[20405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376298, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:58.937829 20512 init.cc:377] Logged in from keytab as kudu/127.2.74.193@KRBTEST.COM (short username kudu)
I20260925 22:44:58.938710 20512 webserver.cc:460] Webserver started at http://127.2.74.193:34989/ using document root <none> and password file <none>
I20260925 22:44:58.939324 20512 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:58.939391 20512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:58.939558 20512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:58.941591 20512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-0/data/instance:
uuid: "42945f8e373b4964940e41dfdc6c92e1"
format_stamp: "Formatted at 2026-09-25 22:44:58 on dist-test-slave-csnl"
server_key: "1181b645914a249070f0b6adff12a657"
server_key_iv: "6662b1c052ba832d275e3514d2a04108"
server_key_version: "encryptionkey@0"
I20260925 22:44:58.942111 20512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance:
uuid: "42945f8e373b4964940e41dfdc6c92e1"
format_stamp: "Formatted at 2026-09-25 22:44:58 on dist-test-slave-csnl"
server_key: "1181b645914a249070f0b6adff12a657"
server_key_iv: "6662b1c052ba832d275e3514d2a04108"
server_key_version: "encryptionkey@0"
I20260925 22:44:58.946190 20512 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 22:44:58.948698 20527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:58.950003 20512 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:44:58.950179 20512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "42945f8e373b4964940e41dfdc6c92e1"
format_stamp: "Formatted at 2026-09-25 22:44:58 on dist-test-slave-csnl"
server_key: "1181b645914a249070f0b6adff12a657"
server_key_iv: "6662b1c052ba832d275e3514d2a04108"
server_key_version: "encryptionkey@0"
I20260925 22:44:58.950304 20512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:58.976892 20512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:58.980510 20512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:58.980731 20512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:58.981381 20512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:58.982262 20512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:58.982337 20512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:58.982414 20512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:58.982457 20512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:58.991080 20512 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:42131
I20260925 22:44:58.991098 20640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:42131 every 8 connection(s)
I20260925 22:44:58.992256 20512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:44:58.995260 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 20512
I20260925 22:44:58.995409 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance
I20260925 22:44:58.995790 2347 external_mini_cluster.cc:1428] Setting key 3bab9c6fbb600eba5ada9c87d5388c7d
Sep 25 22:44:59 dist-test-slave-csnl krb5kdc[20405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376298, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.74.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.194@KRBTEST.COM" created.
I20260925 22:44:59.009431 20489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:58.994074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:33013 (local address 127.2.74.254:33599)
0925 22:44:58.994218 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:58.994223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:58.995216 (+ 993us) server_negotiation.cc:407] Connection header received
0925 22:44:58.996772 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:58.996775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:58.996825 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:58.996910 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:58.998570 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.999229 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.000992 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.001464 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.001595 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.005006 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:59.005010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:59.005012 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:59.005043 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:59.006788 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:59.007470 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:59.007477 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:59.007482 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:59.007603 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:59.007996 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:59.007999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:59.008000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:59.008228 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:59.008375 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:59.009057 (+ 682us) server_negotiation.cc:299] Negotiation successful
0925 22:44:59.009260 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:44:59.010426 20643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:58.994459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33599 (local address 127.2.74.193:33013)
0925 22:44:58.995017 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:58.995080 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:58.996560 (+ 1480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:58.997088 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:58.997095 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:58.997592 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:58.998367 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:58.998393 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.999372 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.999377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:59.000845 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:59.000858 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.001647 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.001650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:59.001806 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.002543 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:59.002571 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:59.004749 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:59.006921 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:59.006927 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:59.006943 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:59.007300 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:59.007701 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:59.007704 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:59.007705 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:59.007851 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:59.008405 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:59.008416 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:59.008717 (+ 301us) client_negotiation.cc:769] Sending connection context
0925 22:44:59.008985 (+ 268us) client_negotiation.cc:240] Negotiation successful
0925 22:44:59.009408 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":125,"threads_started":1}
I20260925 22:44:59.012156 20641 heartbeater.cc:347] Connected to a master server at 127.2.74.254:33599
I20260925 22:44:59.012542 20641 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:59.013137 20641 heartbeater.cc:511] Master 127.2.74.254:33599 requested a full tablet report, sending...
I20260925 22:44:59.015144 20450 ts_manager.cc:194] Registered new tserver with Master: 42945f8e373b4964940e41dfdc6c92e1 (127.2.74.193:42131)
I20260925 22:44:59.016355 20450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.193@KRBTEST.COM'} at 127.2.74.193:33013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
I20260925 22:44:59.028271 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:0
--local_ip_for_outbound_sockets=127.2.74.194
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:33599
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:44061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:59.146238 20646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:59.146555 20646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:59.146601 20646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:59.150878 20646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:59.150996 20646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:59.151119 20646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:44:59.157099 20646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:44061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:33599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:44:59.158346 20646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:59.159585 20646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:59.170069 20652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:59.170078 20651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:59.170150 20646 server_base.cc:1034] running on GCE node
W20260925 22:44:59.170076 20654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:59.170768 20646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:59.171525 20646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:59.172767 20646 hybrid_clock.cc:648] HybridClock initialized: now 1790376299172737 us; error 44 us; skew 500 ppm
Sep 25 22:44:59 dist-test-slave-csnl krb5kdc[20405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376299, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:59.176378 20646 init.cc:377] Logged in from keytab as kudu/127.2.74.194@KRBTEST.COM (short username kudu)
I20260925 22:44:59.177771 20646 webserver.cc:460] Webserver started at http://127.2.74.194:36677/ using document root <none> and password file <none>
I20260925 22:44:59.178578 20646 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:59.178689 20646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:59.179239 20646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:59.182415 20646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-1/data/instance:
uuid: "8c201ff8dbb24466a0fb6da3066d689f"
format_stamp: "Formatted at 2026-09-25 22:44:59 on dist-test-slave-csnl"
server_key: "da1bc13c7dd13bd3a38118bea75e0ab9"
server_key_iv: "ca704245069ffcae9ae9b6bb19aff4cc"
server_key_version: "encryptionkey@0"
I20260925 22:44:59.183249 20646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance:
uuid: "8c201ff8dbb24466a0fb6da3066d689f"
format_stamp: "Formatted at 2026-09-25 22:44:59 on dist-test-slave-csnl"
server_key: "da1bc13c7dd13bd3a38118bea75e0ab9"
server_key_iv: "ca704245069ffcae9ae9b6bb19aff4cc"
server_key_version: "encryptionkey@0"
I20260925 22:44:59.188951 20646 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 22:44:59.192022 20661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:59.193441 20646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:44:59.193708 20646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "8c201ff8dbb24466a0fb6da3066d689f"
format_stamp: "Formatted at 2026-09-25 22:44:59 on dist-test-slave-csnl"
server_key: "da1bc13c7dd13bd3a38118bea75e0ab9"
server_key_iv: "ca704245069ffcae9ae9b6bb19aff4cc"
server_key_version: "encryptionkey@0"
I20260925 22:44:59.193848 20646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:59.226553 20646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:59.230662 20646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:59.230892 20646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:59.232043 20646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:59.233433 20646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:59.233496 20646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:59.233556 20646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:59.233582 20646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:59.247231 20646 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:43173
I20260925 22:44:59.247254 20774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:43173 every 8 connection(s)
I20260925 22:44:59.248528 20646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:44:59.256451 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 20646
I20260925 22:44:59.256593 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance
I20260925 22:44:59.256951 2347 external_mini_cluster.cc:1428] Setting key f031eb1657fb11f989ab32948d742093
Sep 25 22:44:59 dist-test-slave-csnl krb5kdc[20405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376299, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:44:59.266359 20489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.250587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:55821 (local address 127.2.74.254:33599)
0925 22:44:59.252851 (+ 2264us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:59.252857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:59.252878 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:44:59.253501 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:59.253507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:59.253577 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:59.253686 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:59.255342 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.256128 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.257844 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.258182 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.258313 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.261604 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:59.261608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:59.261611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:59.261650 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:59.263536 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:59.264202 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:59.264205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:59.264207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:59.264306 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:59.264787 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:59.264793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:59.264795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:59.265101 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:59.265202 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:59.265997 (+ 795us) server_negotiation.cc:299] Negotiation successful
0925 22:44:59.266171 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2144}
I20260925 22:44:59.268263 20777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.251113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33599 (local address 127.2.74.194:55821)
0925 22:44:59.251803 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:59.251876 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:59.253200 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:59.253924 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:59.253933 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:59.254378 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:59.255131 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:59.255153 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.256328 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.256334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:59.257740 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:59.257752 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.258360 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.258364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:59.258580 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.259306 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:59.259324 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:59.261402 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:59.263693 (+ 2291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:59.263702 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:59.263721 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:59.264081 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:59.264423 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:59.264430 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:59.264434 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:59.264652 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:59.265283 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:59.265292 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:59.265676 (+ 384us) client_negotiation.cc:769] Sending connection context
0925 22:44:59.266579 (+ 903us) client_negotiation.cc:240] Negotiation successful
0925 22:44:59.267020 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":147,"threads_started":1}
WARNING: no policy specified for kudu/127.2.74.195@KRBTEST.COM; defaulting to no policy
I20260925 22:44:59.269749 20775 heartbeater.cc:347] Connected to a master server at 127.2.74.254:33599
I20260925 22:44:59.270021 20775 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:59.270668 20775 heartbeater.cc:511] Master 127.2.74.254:33599 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.195@KRBTEST.COM" created.
I20260925 22:44:59.272032 20450 ts_manager.cc:194] Registered new tserver with Master: 8c201ff8dbb24466a0fb6da3066d689f (127.2.74.194:43173)
I20260925 22:44:59.272903 20450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.194@KRBTEST.COM'} at 127.2.74.194:55821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
I20260925 22:44:59.290627 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:0
--local_ip_for_outbound_sockets=127.2.74.195
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:33599
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:44061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:59.412626 20780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:59.412943 20780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:59.412988 20780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:59.417677 20780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:59.417758 20780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:59.417822 20780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:44:59.423686 20780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:44061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:33599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:44:59.424932 20780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:59.426069 20780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:59.437026 20788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:59.437026 20786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:59.437027 20785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:59.437307 20780 server_base.cc:1034] running on GCE node
I20260925 22:44:59.437826 20780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:59.438596 20780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:59.439805 20780 hybrid_clock.cc:648] HybridClock initialized: now 1790376299439747 us; error 76 us; skew 500 ppm
Sep 25 22:44:59 dist-test-slave-csnl krb5kdc[20405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376299, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:59.442883 20780 init.cc:377] Logged in from keytab as kudu/127.2.74.195@KRBTEST.COM (short username kudu)
I20260925 22:44:59.444012 20780 webserver.cc:460] Webserver started at http://127.2.74.195:41791/ using document root <none> and password file <none>
I20260925 22:44:59.444609 20780 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:59.444677 20780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:59.444854 20780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:59.447095 20780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-2/data/instance:
uuid: "c72363cf8fe94065879f9a474e21b950"
format_stamp: "Formatted at 2026-09-25 22:44:59 on dist-test-slave-csnl"
server_key: "9a59030381ff6a8e94dc8561798c686a"
server_key_iv: "d981f158d47e04e9ad6095d402a649d7"
server_key_version: "encryptionkey@0"
I20260925 22:44:59.447681 20780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance:
uuid: "c72363cf8fe94065879f9a474e21b950"
format_stamp: "Formatted at 2026-09-25 22:44:59 on dist-test-slave-csnl"
server_key: "9a59030381ff6a8e94dc8561798c686a"
server_key_iv: "d981f158d47e04e9ad6095d402a649d7"
server_key_version: "encryptionkey@0"
I20260925 22:44:59.451951 20780 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 22:44:59.454936 20795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:59.456398 20780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:44:59.456669 20780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "c72363cf8fe94065879f9a474e21b950"
format_stamp: "Formatted at 2026-09-25 22:44:59 on dist-test-slave-csnl"
server_key: "9a59030381ff6a8e94dc8561798c686a"
server_key_iv: "d981f158d47e04e9ad6095d402a649d7"
server_key_version: "encryptionkey@0"
I20260925 22:44:59.456813 20780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:59.467878 20780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:59.471179 20780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:59.471386 20780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:59.472072 20780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:59.472937 20780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:59.473011 20780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:59.473093 20780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:59.473145 20780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:59.482884 20780 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:33225
I20260925 22:44:59.483011 20908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:33225 every 8 connection(s)
I20260925 22:44:59.483934 20780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:44:59.488083 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 20780
I20260925 22:44:59.488262 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance
I20260925 22:44:59.488673 2347 external_mini_cluster.cc:1428] Setting key b0732929abd540a4bef6af4b53a64240
Sep 25 22:44:59 dist-test-slave-csnl krb5kdc[20405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376299, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:44:59.499563 20489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.485950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:52515 (local address 127.2.74.254:33599)
0925 22:44:59.486097 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:59.486101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:59.486984 (+ 883us) server_negotiation.cc:407] Connection header received
0925 22:44:59.488373 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:59.488378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:59.488449 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:59.488554 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:59.490040 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.490843 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.492211 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.492453 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.492553 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.495499 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:59.495502 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:59.495505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:59.495536 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:59.496986 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:59.497745 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:59.497749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:59.497750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:59.497842 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:59.498205 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:59.498208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:59.498209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:59.498408 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:59.498506 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:59.499280 (+ 774us) server_negotiation.cc:299] Negotiation successful
0925 22:44:59.499402 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:44:59.500430 20911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.486290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33599 (local address 127.2.74.195:52515)
0925 22:44:59.486778 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:59.486825 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:59.488091 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:59.488754 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:59.488765 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:59.489221 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:59.489884 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:59.489900 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.491039 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.491043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:59.492037 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:59.492048 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.492612 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.492615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:59.492784 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.493476 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:59.493495 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:59.495281 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:59.497194 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:59.497200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:59.497216 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:59.497581 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:59.497944 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:59.497946 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:59.497948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:59.498107 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:59.498578 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:59.498588 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:59.498876 (+ 288us) client_negotiation.cc:769] Sending connection context
0925 22:44:59.499148 (+ 272us) client_negotiation.cc:240] Negotiation successful
0925 22:44:59.499492 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":126,"threads_started":1}
I20260925 22:44:59.501921 20909 heartbeater.cc:347] Connected to a master server at 127.2.74.254:33599
I20260925 22:44:59.502229 20909 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:59.502862 20909 heartbeater.cc:511] Master 127.2.74.254:33599 requested a full tablet report, sending...
I20260925 22:44:59.504084 20450 ts_manager.cc:194] Registered new tserver with Master: c72363cf8fe94065879f9a474e21b950 (127.2.74.195:33225)
I20260925 22:44:59.504630 20450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.195@KRBTEST.COM'} at 127.2.74.195:52515
I20260925 22:44:59.515571 2347 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:44:59.526154 20489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.517820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41122 (local address 127.2.74.254:33599)
0925 22:44:59.517960 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:59.517963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:59.517980 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:44:59.518152 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:59.518156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:59.518212 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:59.518313 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:59.519193 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.519748 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.520673 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.520898 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.520988 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.522754 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:59.522756 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:59.522758 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:59.522780 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:59.524175 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:59.524707 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:59.524710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:59.524711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:59.524791 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:59.525203 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:59.525206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:59.525207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:59.525447 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:59.525583 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:59.525906 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 22:44:59.526013 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:44:59.530025 20450 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:44:59.533305 20450 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:44:59.548513 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.542381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:43173 (local address 127.0.0.1:37020)
0925 22:44:59.542835 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:59.542895 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:59.543094 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:59.543850 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:59.543858 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:59.543883 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:59.544457 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:59.544472 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.545678 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.545683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:59.547023 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:59.547034 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.547924 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.547928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:59.548053 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.548106 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 22:44:59.548220 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 22:44:59.548348 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":145,"threads_started":1}
I20260925 22:44:59.548528 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.542591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37020 (local address 127.2.74.194:43173)
0925 22:44:59.543001 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:59.543015 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:59.543040 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:44:59.543312 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:59.543319 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:59.543553 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:59.543733 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:59.544592 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.545540 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.547163 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.547803 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.547912 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.548064 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:59.548271 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 22:44:59.548374 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":118,"threads_started":1}
I20260925 22:44:59.549348 20922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.543317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:42131 (local address 127.0.0.1:36718)
0925 22:44:59.543596 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:59.543609 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:59.543710 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:59.544824 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:59.544827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:59.544847 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:59.544954 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:59.544961 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.546370 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.546374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:59.547651 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:59.547662 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.548946 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.548950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:59.549046 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.549069 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:44:59.549148 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 22:44:59.549218 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":100,"threads_started":1}
I20260925 22:44:59.549572 20924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.544045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36718 (local address 127.2.74.193:42131)
0925 22:44:59.544368 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:59.544375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:59.544400 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:44:59.544477 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:59.544482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:59.544684 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:59.544837 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:59.545077 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.546239 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.547791 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.548599 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.548723 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.548876 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:59.549208 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 22:44:59.549311 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":98,"threads_started":1}
I20260925 22:44:59.549604 20923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.543629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:33225 (local address 127.0.0.1:36528)
0925 22:44:59.544146 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:59.544157 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:59.544276 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:59.544924 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:59.544926 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:59.544945 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:59.545064 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:59.545071 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.546546 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.546549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:59.547718 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:59.547729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.549233 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.549237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:59.549324 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.549346 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:44:59.549394 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 22:44:59.549464 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":83,"threads_started":1}
I20260925 22:44:59.550146 20925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.544124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36528 (local address 127.2.74.195:33225)
0925 22:44:59.544518 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:59.544528 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:59.544548 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:44:59.544623 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:59.544627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:59.544779 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:59.545214 (+ 435us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:59.545362 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.546413 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.547841 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.549075 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.549827 (+ 752us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.549929 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:59.549973 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 22:44:59.550038 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":122,"threads_started":1}
I20260925 22:44:59.551964 20693 tablet_service.cc:1467] Processing CreateTablet for tablet 38346a3828254285abdc396bf9d90246 (DEFAULT_TABLE table=test-table [id=a45aba37b5cb4cd28ce901223ab9d942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:59.553216 20693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38346a3828254285abdc396bf9d90246. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:59.553035 20575 tablet_service.cc:1467] Processing CreateTablet for tablet 38346a3828254285abdc396bf9d90246 (DEFAULT_TABLE table=test-table [id=a45aba37b5cb4cd28ce901223ab9d942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:59.553098 20843 tablet_service.cc:1467] Processing CreateTablet for tablet 38346a3828254285abdc396bf9d90246 (DEFAULT_TABLE table=test-table [id=a45aba37b5cb4cd28ce901223ab9d942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:59.554188 20575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38346a3828254285abdc396bf9d90246. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:59.554203 20843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38346a3828254285abdc396bf9d90246. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:59.560684 20926 tablet_bootstrap.cc:492] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1: Bootstrap starting.
I20260925 22:44:59.561830 20928 tablet_bootstrap.cc:492] T 38346a3828254285abdc396bf9d90246 P c72363cf8fe94065879f9a474e21b950: Bootstrap starting.
I20260925 22:44:59.563285 20926 tablet_bootstrap.cc:654] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:59.563342 20927 tablet_bootstrap.cc:492] T 38346a3828254285abdc396bf9d90246 P 8c201ff8dbb24466a0fb6da3066d689f: Bootstrap starting.
I20260925 22:44:59.564205 20928 tablet_bootstrap.cc:654] T 38346a3828254285abdc396bf9d90246 P c72363cf8fe94065879f9a474e21b950: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:59.564256 20926 log.cc:824] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:59.565194 20928 log.cc:824] T 38346a3828254285abdc396bf9d90246 P c72363cf8fe94065879f9a474e21b950: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:59.565948 20927 tablet_bootstrap.cc:654] T 38346a3828254285abdc396bf9d90246 P 8c201ff8dbb24466a0fb6da3066d689f: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:59.566740 20927 log.cc:824] T 38346a3828254285abdc396bf9d90246 P 8c201ff8dbb24466a0fb6da3066d689f: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:59.566893 20926 tablet_bootstrap.cc:492] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1: No bootstrap required, opened a new log
I20260925 22:44:59.567138 20926 ts_tablet_manager.cc:1397] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:44:59.567330 20928 tablet_bootstrap.cc:492] T 38346a3828254285abdc396bf9d90246 P c72363cf8fe94065879f9a474e21b950: No bootstrap required, opened a new log
I20260925 22:44:59.567524 20928 ts_tablet_manager.cc:1397] T 38346a3828254285abdc396bf9d90246 P c72363cf8fe94065879f9a474e21b950: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:44:59.568579 20927 tablet_bootstrap.cc:492] T 38346a3828254285abdc396bf9d90246 P 8c201ff8dbb24466a0fb6da3066d689f: No bootstrap required, opened a new log
I20260925 22:44:59.568801 20927 ts_tablet_manager.cc:1397] T 38346a3828254285abdc396bf9d90246 P 8c201ff8dbb24466a0fb6da3066d689f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:44:59.571936 20926 raft_consensus.cc:348] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c201ff8dbb24466a0fb6da3066d689f" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 43173 } } peers { permanent_uuid: "42945f8e373b4964940e41dfdc6c92e1" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 42131 } } peers { permanent_uuid: "c72363cf8fe94065879f9a474e21b950" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33225 } }
I20260925 22:44:59.571802 20928 raft_consensus.cc:348] T 38346a3828254285abdc396bf9d90246 P c72363cf8fe94065879f9a474e21b950 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c201ff8dbb24466a0fb6da3066d689f" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 43173 } } peers { permanent_uuid: "42945f8e373b4964940e41dfdc6c92e1" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 42131 } } peers { permanent_uuid: "c72363cf8fe94065879f9a474e21b950" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33225 } }
I20260925 22:44:59.572190 20928 raft_consensus.cc:374] T 38346a3828254285abdc396bf9d90246 P c72363cf8fe94065879f9a474e21b950 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:59.572189 20926 raft_consensus.cc:374] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:59.572258 20928 raft_consensus.cc:729] T 38346a3828254285abdc396bf9d90246 P c72363cf8fe94065879f9a474e21b950 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c72363cf8fe94065879f9a474e21b950, State: Initialized, Role: FOLLOWER
I20260925 22:44:59.572259 20926 raft_consensus.cc:729] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42945f8e373b4964940e41dfdc6c92e1, State: Initialized, Role: FOLLOWER
I20260925 22:44:59.572278 20927 raft_consensus.cc:348] T 38346a3828254285abdc396bf9d90246 P 8c201ff8dbb24466a0fb6da3066d689f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c201ff8dbb24466a0fb6da3066d689f" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 43173 } } peers { permanent_uuid: "42945f8e373b4964940e41dfdc6c92e1" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 42131 } } peers { permanent_uuid: "c72363cf8fe94065879f9a474e21b950" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33225 } }
I20260925 22:44:59.572667 20927 raft_consensus.cc:374] T 38346a3828254285abdc396bf9d90246 P 8c201ff8dbb24466a0fb6da3066d689f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:59.572736 20927 raft_consensus.cc:729] T 38346a3828254285abdc396bf9d90246 P 8c201ff8dbb24466a0fb6da3066d689f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c201ff8dbb24466a0fb6da3066d689f, State: Initialized, Role: FOLLOWER
I20260925 22:44:59.572783 20926 consensus_queue.cc:260] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c201ff8dbb24466a0fb6da3066d689f" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 43173 } } peers { permanent_uuid: "42945f8e373b4964940e41dfdc6c92e1" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 42131 } } peers { permanent_uuid: "c72363cf8fe94065879f9a474e21b950" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33225 } }
I20260925 22:44:59.572783 20928 consensus_queue.cc:260] T 38346a3828254285abdc396bf9d90246 P c72363cf8fe94065879f9a474e21b950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c201ff8dbb24466a0fb6da3066d689f" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 43173 } } peers { permanent_uuid: "42945f8e373b4964940e41dfdc6c92e1" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 42131 } } peers { permanent_uuid: "c72363cf8fe94065879f9a474e21b950" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33225 } }
I20260925 22:44:59.573297 20927 consensus_queue.cc:260] T 38346a3828254285abdc396bf9d90246 P 8c201ff8dbb24466a0fb6da3066d689f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c201ff8dbb24466a0fb6da3066d689f" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 43173 } } peers { permanent_uuid: "42945f8e373b4964940e41dfdc6c92e1" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 42131 } } peers { permanent_uuid: "c72363cf8fe94065879f9a474e21b950" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33225 } }
I20260925 22:44:59.573741 20909 heartbeater.cc:503] Master 127.2.74.254:33599 was elected leader, sending a full tablet report...
I20260925 22:44:59.574051 20926 ts_tablet_manager.cc:1428] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:44:59.574051 20928 ts_tablet_manager.cc:1428] T 38346a3828254285abdc396bf9d90246 P c72363cf8fe94065879f9a474e21b950: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:44:59.574311 20927 ts_tablet_manager.cc:1428] T 38346a3828254285abdc396bf9d90246 P 8c201ff8dbb24466a0fb6da3066d689f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 22:44:59.574427 20641 heartbeater.cc:503] Master 127.2.74.254:33599 was elected leader, sending a full tablet report...
I20260925 22:44:59.574656 20775 heartbeater.cc:503] Master 127.2.74.254:33599 was elected leader, sending a full tablet report...
W20260925 22:44:59.735064 20910 tablet.cc:2369] T 38346a3828254285abdc396bf9d90246 P c72363cf8fe94065879f9a474e21b950: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:44:59.743652 20642 tablet.cc:2369] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:44:59.749897 20776 tablet.cc:2369] T 38346a3828254285abdc396bf9d90246 P 8c201ff8dbb24466a0fb6da3066d689f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:44:59.896838 20933 raft_consensus.cc:482] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:44:59.897080 20933 raft_consensus.cc:504] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c201ff8dbb24466a0fb6da3066d689f" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 43173 } } peers { permanent_uuid: "42945f8e373b4964940e41dfdc6c92e1" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 42131 } } peers { permanent_uuid: "c72363cf8fe94065879f9a474e21b950" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33225 } }
I20260925 22:44:59.898425 20933 leader_election.cc:288] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c201ff8dbb24466a0fb6da3066d689f (127.2.74.194:43173), c72363cf8fe94065879f9a474e21b950 (127.2.74.195:33225)
I20260925 22:44:59.904327 20925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.899352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:59439 (local address 127.2.74.195:33225)
0925 22:44:59.899484 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:59.899488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:59.899709 (+ 221us) server_negotiation.cc:407] Connection header received
0925 22:44:59.899769 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:59.899771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:59.899814 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:59.899874 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:59.900146 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.900868 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.902790 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.903470 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.903614 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.903670 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:59.904100 (+ 430us) server_negotiation.cc:299] Negotiation successful
0925 22:44:59.904180 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:44:59.904327 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.898804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:38975 (local address 127.2.74.194:43173)
0925 22:44:59.898991 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:59.898995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:59.899260 (+ 265us) server_negotiation.cc:407] Connection header received
0925 22:44:59.899424 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:59.899427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:59.899485 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:59.899577 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:59.899981 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.900797 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.902790 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.903470 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.903614 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.903670 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:59.904100 (+ 430us) server_negotiation.cc:299] Negotiation successful
0925 22:44:59.904187 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 22:44:59.904407 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.898725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:43173 (local address 127.2.74.193:38975)
0925 22:44:59.899107 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:59.899121 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:59.899290 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:59.899637 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:59.899643 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:59.899668 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:59.899829 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:59.899835 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.900949 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.900954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:59.902616 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:59.902632 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.903623 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.903627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:59.903865 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.903961 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 22:44:59.904056 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 22:44:59.904210 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":290,"thread_start_us":133,"threads_started":1}
I20260925 22:44:59.904623 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.899203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:33225 (local address 127.2.74.193:59439)
0925 22:44:59.899595 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:59.899609 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:59.899701 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:59.899919 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:59.899924 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:59.899943 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:59.900038 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:59.900042 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.900975 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.900981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:59.902616 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:59.902632 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.903623 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.903627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:59.903863 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.903961 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 22:44:59.904056 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 22:44:59.904210 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":242,"thread_start_us":143,"threads_started":1}
I20260925 22:44:59.905230 20729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38346a3828254285abdc396bf9d90246" candidate_uuid: "42945f8e373b4964940e41dfdc6c92e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c201ff8dbb24466a0fb6da3066d689f" is_pre_election: true
I20260925 22:44:59.905428 20863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38346a3828254285abdc396bf9d90246" candidate_uuid: "42945f8e373b4964940e41dfdc6c92e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c72363cf8fe94065879f9a474e21b950" is_pre_election: true
I20260925 22:44:59.905608 20729 raft_consensus.cc:2454] T 38346a3828254285abdc396bf9d90246 P 8c201ff8dbb24466a0fb6da3066d689f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42945f8e373b4964940e41dfdc6c92e1 in term 0.
I20260925 22:44:59.905742 20863 raft_consensus.cc:2454] T 38346a3828254285abdc396bf9d90246 P c72363cf8fe94065879f9a474e21b950 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42945f8e373b4964940e41dfdc6c92e1 in term 0.
I20260925 22:44:59.906219 20529 leader_election.cc:302] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42945f8e373b4964940e41dfdc6c92e1, 8c201ff8dbb24466a0fb6da3066d689f; no voters:
I20260925 22:44:59.906553 20933 raft_consensus.cc:2781] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:44:59.906637 20933 raft_consensus.cc:482] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:44:59.906697 20933 raft_consensus.cc:3037] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:59.907828 20933 raft_consensus.cc:504] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c201ff8dbb24466a0fb6da3066d689f" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 43173 } } peers { permanent_uuid: "42945f8e373b4964940e41dfdc6c92e1" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 42131 } } peers { permanent_uuid: "c72363cf8fe94065879f9a474e21b950" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33225 } }
I20260925 22:44:59.908224 20933 leader_election.cc:288] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1 [CANDIDATE]: Term 1 election: Requested vote from peers 8c201ff8dbb24466a0fb6da3066d689f (127.2.74.194:43173), c72363cf8fe94065879f9a474e21b950 (127.2.74.195:33225)
I20260925 22:44:59.908806 20863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38346a3828254285abdc396bf9d90246" candidate_uuid: "42945f8e373b4964940e41dfdc6c92e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c72363cf8fe94065879f9a474e21b950"
I20260925 22:44:59.908852 20729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38346a3828254285abdc396bf9d90246" candidate_uuid: "42945f8e373b4964940e41dfdc6c92e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c201ff8dbb24466a0fb6da3066d689f"
I20260925 22:44:59.908928 20863 raft_consensus.cc:3037] T 38346a3828254285abdc396bf9d90246 P c72363cf8fe94065879f9a474e21b950 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:59.909016 20729 raft_consensus.cc:3037] T 38346a3828254285abdc396bf9d90246 P 8c201ff8dbb24466a0fb6da3066d689f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:59.910123 20729 raft_consensus.cc:2454] T 38346a3828254285abdc396bf9d90246 P 8c201ff8dbb24466a0fb6da3066d689f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42945f8e373b4964940e41dfdc6c92e1 in term 1.
I20260925 22:44:59.910132 20863 raft_consensus.cc:2454] T 38346a3828254285abdc396bf9d90246 P c72363cf8fe94065879f9a474e21b950 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42945f8e373b4964940e41dfdc6c92e1 in term 1.
I20260925 22:44:59.910557 20529 leader_election.cc:302] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42945f8e373b4964940e41dfdc6c92e1, c72363cf8fe94065879f9a474e21b950; no voters:
I20260925 22:44:59.910778 20933 raft_consensus.cc:2781] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:59.911121 20933 raft_consensus.cc:686] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1 [term 1 LEADER]: Becoming Leader. State: Replica: 42945f8e373b4964940e41dfdc6c92e1, State: Running, Role: LEADER
I20260925 22:44:59.911571 20933 consensus_queue.cc:237] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c201ff8dbb24466a0fb6da3066d689f" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 43173 } } peers { permanent_uuid: "42945f8e373b4964940e41dfdc6c92e1" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 42131 } } peers { permanent_uuid: "c72363cf8fe94065879f9a474e21b950" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33225 } }
I20260925 22:44:59.915376 20450 catalog_manager.cc:5495] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42945f8e373b4964940e41dfdc6c92e1 (127.2.74.193). New cstate: current_term: 1 leader_uuid: "42945f8e373b4964940e41dfdc6c92e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c201ff8dbb24466a0fb6da3066d689f" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 43173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42945f8e373b4964940e41dfdc6c92e1" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 42131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c72363cf8fe94065879f9a474e21b950" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33225 } health_report { overall_health: UNKNOWN } } }
I20260925 22:45:00.010217 20924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:00.006454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36720 (local address 127.2.74.193:42131)
0925 22:45:00.006591 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:00.006595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:00.006609 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:45:00.006691 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:00.006694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:00.006752 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:00.006855 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:45:00.007181 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:00.007707 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:00.008634 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:00.008863 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:00.008953 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:00.009325 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:45:00.009771 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:45:00.009897 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:00.010043 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 22:45:00.010092 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:45:00.017434 20863 raft_consensus.cc:1261] T 38346a3828254285abdc396bf9d90246 P c72363cf8fe94065879f9a474e21b950 [term 1 FOLLOWER]: Refusing update from remote peer 42945f8e373b4964940e41dfdc6c92e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:00.017434 20729 raft_consensus.cc:1261] T 38346a3828254285abdc396bf9d90246 P 8c201ff8dbb24466a0fb6da3066d689f [term 1 FOLLOWER]: Refusing update from remote peer 42945f8e373b4964940e41dfdc6c92e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:00.018275 20937 consensus_queue.cc:1035] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c72363cf8fe94065879f9a474e21b950" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:00.018532 20933 consensus_queue.cc:1035] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c201ff8dbb24466a0fb6da3066d689f" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 43173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:00.026933 20940 mvcc.cc:205] Tried to move back new op lower bound from 7333381324861382656 to 7333381324441018368. Current Snapshot: MvccSnapshot[applied={T|T < 7333381324861382656}]
I20260925 22:45:00.036914 20447 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:45:00.037107 20447 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:45:00.040052 20447 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7427acd1a97d462f843784ff57bb1eb2: Sending DeleteTablet for 3 replicas of tablet 38346a3828254285abdc396bf9d90246
I20260925 22:45:00.041050 20693 tablet_service.cc:1514] Processing DeleteTablet for tablet 38346a3828254285abdc396bf9d90246 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:00 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:37020
I20260925 22:45:00.041342 20843 tablet_service.cc:1514] Processing DeleteTablet for tablet 38346a3828254285abdc396bf9d90246 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:00 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:36528
I20260925 22:45:00.041368 20575 tablet_service.cc:1514] Processing DeleteTablet for tablet 38346a3828254285abdc396bf9d90246 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:00 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:36718
I20260925 22:45:00.041900 20951 tablet_replica.cc:330] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1: stopping tablet replica
I20260925 22:45:00.042013 20950 tablet_replica.cc:330] T 38346a3828254285abdc396bf9d90246 P 8c201ff8dbb24466a0fb6da3066d689f: stopping tablet replica
I20260925 22:45:00.042300 20951 raft_consensus.cc:2229] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:45:00.042400 20950 raft_consensus.cc:2229] T 38346a3828254285abdc396bf9d90246 P 8c201ff8dbb24466a0fb6da3066d689f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:45:00.042599 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 20512
I20260925 22:45:00.042780 20951 raft_consensus.cc:2258] T 38346a3828254285abdc396bf9d90246 P 42945f8e373b4964940e41dfdc6c92e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:45:00.042815 20950 raft_consensus.cc:2258] T 38346a3828254285abdc396bf9d90246 P 8c201ff8dbb24466a0fb6da3066d689f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:45:00.043728 20952 tablet_replica.cc:330] T 38346a3828254285abdc396bf9d90246 P c72363cf8fe94065879f9a474e21b950: stopping tablet replica
I20260925 22:45:00.044067 20952 raft_consensus.cc:2229] T 38346a3828254285abdc396bf9d90246 P c72363cf8fe94065879f9a474e21b950 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:45:00.044426 20952 raft_consensus.cc:2258] T 38346a3828254285abdc396bf9d90246 P c72363cf8fe94065879f9a474e21b950 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:45:00.045022 20950 ts_tablet_manager.cc:1905] T 38346a3828254285abdc396bf9d90246 P 8c201ff8dbb24466a0fb6da3066d689f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:45:00.046042 20952 ts_tablet_manager.cc:1905] T 38346a3828254285abdc396bf9d90246 P c72363cf8fe94065879f9a474e21b950: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:45:00.048095 20950 ts_tablet_manager.cc:1918] T 38346a3828254285abdc396bf9d90246 P 8c201ff8dbb24466a0fb6da3066d689f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:45:00.048224 20950 log.cc:1196] T 38346a3828254285abdc396bf9d90246 P 8c201ff8dbb24466a0fb6da3066d689f: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-1/wal/wals/38346a3828254285abdc396bf9d90246
I20260925 22:45:00.048550 20950 ts_tablet_manager.cc:1939] T 38346a3828254285abdc396bf9d90246 P 8c201ff8dbb24466a0fb6da3066d689f: Deleting consensus metadata
I20260925 22:45:00.049084 20952 ts_tablet_manager.cc:1918] T 38346a3828254285abdc396bf9d90246 P c72363cf8fe94065879f9a474e21b950: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:45:00.049187 20952 log.cc:1196] T 38346a3828254285abdc396bf9d90246 P c72363cf8fe94065879f9a474e21b950: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376047580784-2347-0/minicluster-data/ts-2/wal/wals/38346a3828254285abdc396bf9d90246
I20260925 22:45:00.049489 20952 ts_tablet_manager.cc:1939] T 38346a3828254285abdc396bf9d90246 P c72363cf8fe94065879f9a474e21b950: Deleting consensus metadata
I20260925 22:45:00.050140 20435 catalog_manager.cc:4841] TS 8c201ff8dbb24466a0fb6da3066d689f (127.2.74.194:43173): tablet 38346a3828254285abdc396bf9d90246 (table test-table [id=a45aba37b5cb4cd28ce901223ab9d942]) successfully deleted
I20260925 22:45:00.051286 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 20646
W20260925 22:45:00.051321 20435 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.74.193:42131 (error 108)
I20260925 22:45:00.051842 20435 catalog_manager.cc:4841] TS c72363cf8fe94065879f9a474e21b950 (127.2.74.195:33225): tablet 38346a3828254285abdc396bf9d90246 (table test-table [id=a45aba37b5cb4cd28ce901223ab9d942]) successfully deleted
I20260925 22:45:00.053661 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:00.053005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:42131 (local address 127.0.0.1:36722)
0925 22:45:00.053426 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:00.053522 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.193:42131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":83,"threads_started":1}
W20260925 22:45:00.053885 20435 catalog_manager.cc:4584] TS 42945f8e373b4964940e41dfdc6c92e1 (127.2.74.193:42131): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 38346a3828254285abdc396bf9d90246: Network error: Client connection negotiation failed: client connection to 127.2.74.193:42131: connect: Connection refused (error 111)
I20260925 22:45:00.057890 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 20780
I20260925 22:45:00.063787 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 20418
2026-09-25T22:45:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2735 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T22:45:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:00Z Disabled control of system clock
I20260925 22:45:00.096357 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:34549
--webserver_interface=127.2.74.254
--webserver_port=0
--builtin_ntp_servers=127.2.74.212:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:34549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:45:00.221182 20958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:00.221495 20958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:00.221542 20958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:00.225629 20958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:00.225708 20958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:00.225723 20958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:00.225742 20958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:00.225757 20958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:00.230792 20958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:34549
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:34549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:00.231966 20958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:00.233441 20958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:45:00.242436 20958 server_base.cc:1034] running on GCE node
W20260925 22:45:00.242393 20964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:00.242383 20963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:00.242383 20966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:00.243278 20958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:00.244422 20958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:00.245595 20958 hybrid_clock.cc:648] HybridClock initialized: now 1790376300245561 us; error 54 us; skew 500 ppm
I20260925 22:45:00.252314 20958 webserver.cc:460] Webserver started at http://127.2.74.254:44505/ using document root <none> and password file <none>
I20260925 22:45:00.253111 20958 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:00.253198 20958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:00.253449 20958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:00.255887 20958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "8d82ad1dab474037a040936bf535dfc0"
format_stamp: "Formatted at 2026-09-25 22:45:00 on dist-test-slave-csnl"
server_key: "00fac6f794cbe99eb69e583de805fa2b"
server_key_iv: "2fa97e3b4adff5c223e691852c1de4bd"
server_key_version: "encryptionkey@0"
I20260925 22:45:00.256553 20958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "8d82ad1dab474037a040936bf535dfc0"
format_stamp: "Formatted at 2026-09-25 22:45:00 on dist-test-slave-csnl"
server_key: "00fac6f794cbe99eb69e583de805fa2b"
server_key_iv: "2fa97e3b4adff5c223e691852c1de4bd"
server_key_version: "encryptionkey@0"
I20260925 22:45:00.260752 20958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 22:45:00.263561 20972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:00.264957 20958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:00.265172 20958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "8d82ad1dab474037a040936bf535dfc0"
format_stamp: "Formatted at 2026-09-25 22:45:00 on dist-test-slave-csnl"
server_key: "00fac6f794cbe99eb69e583de805fa2b"
server_key_iv: "2fa97e3b4adff5c223e691852c1de4bd"
server_key_version: "encryptionkey@0"
I20260925 22:45:00.265349 20958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:00.283093 20958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:00.283926 20958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:00.284170 20958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:00.291937 20958 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:34549
I20260925 22:45:00.291973 21024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:34549 every 8 connection(s)
I20260925 22:45:00.292984 20958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:00.293553 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 20958
I20260925 22:45:00.293697 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:45:00.293953 2347 external_mini_cluster.cc:1428] Setting key 2ad0ecddbee1c3b49cb47217c22fd001
I20260925 22:45:00.297040 21025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:00.304385 21025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d82ad1dab474037a040936bf535dfc0: Bootstrap starting.
I20260925 22:45:00.305095 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:00.295505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37730 (local address 127.2.74.254:34549)
0925 22:45:00.296219 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:00.296239 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:00.296283 (+ 44us) server_negotiation.cc:407] Connection header received
0925 22:45:00.297187 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:00.297211 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:00.297712 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:00.298095 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:00.298568 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:00.300090 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:00.301013 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:00.301696 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:00.301866 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:00.302774 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:00.302781 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:00.302798 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:00.303387 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:00.303461 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:00.303490 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:00.303591 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:00.303788 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 22:45:00.304085 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":121,"threads_started":1}
I20260925 22:45:00.307147 21025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d82ad1dab474037a040936bf535dfc0: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:00.308104 21025 log.cc:824] T 00000000000000000000000000000000 P 8d82ad1dab474037a040936bf535dfc0: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:00.310735 21025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d82ad1dab474037a040936bf535dfc0: No bootstrap required, opened a new log
I20260925 22:45:00.315313 21025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d82ad1dab474037a040936bf535dfc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d82ad1dab474037a040936bf535dfc0" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34549 } }
I20260925 22:45:00.315594 21025 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d82ad1dab474037a040936bf535dfc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:00.315678 21025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d82ad1dab474037a040936bf535dfc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d82ad1dab474037a040936bf535dfc0, State: Initialized, Role: FOLLOWER
I20260925 22:45:00.316182 21025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d82ad1dab474037a040936bf535dfc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d82ad1dab474037a040936bf535dfc0" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34549 } }
I20260925 22:45:00.316380 21025 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d82ad1dab474037a040936bf535dfc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:00.316457 21025 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d82ad1dab474037a040936bf535dfc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:00.316571 21025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d82ad1dab474037a040936bf535dfc0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:00.317624 21025 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d82ad1dab474037a040936bf535dfc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d82ad1dab474037a040936bf535dfc0" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34549 } }
I20260925 22:45:00.318005 21025 leader_election.cc:302] T 00000000000000000000000000000000 P 8d82ad1dab474037a040936bf535dfc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d82ad1dab474037a040936bf535dfc0; no voters:
I20260925 22:45:00.318358 21025 leader_election.cc:288] T 00000000000000000000000000000000 P 8d82ad1dab474037a040936bf535dfc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:00.318568 21030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d82ad1dab474037a040936bf535dfc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:00.318838 21030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d82ad1dab474037a040936bf535dfc0 [term 1 LEADER]: Becoming Leader. State: Replica: 8d82ad1dab474037a040936bf535dfc0, State: Running, Role: LEADER
I20260925 22:45:00.319195 21030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d82ad1dab474037a040936bf535dfc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d82ad1dab474037a040936bf535dfc0" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34549 } }
I20260925 22:45:00.319818 21025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d82ad1dab474037a040936bf535dfc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:00.321039 21031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d82ad1dab474037a040936bf535dfc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d82ad1dab474037a040936bf535dfc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d82ad1dab474037a040936bf535dfc0" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34549 } } }
I20260925 22:45:00.321039 21032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d82ad1dab474037a040936bf535dfc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d82ad1dab474037a040936bf535dfc0. Latest consensus state: current_term: 1 leader_uuid: "8d82ad1dab474037a040936bf535dfc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d82ad1dab474037a040936bf535dfc0" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34549 } } }
I20260925 22:45:00.321203 21031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d82ad1dab474037a040936bf535dfc0 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:00.321203 21032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d82ad1dab474037a040936bf535dfc0 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:00.321732 21039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:00.325057 21039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:00.331102 21039 catalog_manager.cc:1347] Generated new cluster ID: b8654ef88c894993aaba4d5ad6ee320d
I20260925 22:45:00.331200 21039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:00.351608 21039 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:00.352610 21039 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:00.359763 21039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d82ad1dab474037a040936bf535dfc0: Generated new TSK 0
I20260925 22:45:00.360569 21039 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:45:00.369132 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:0
--local_ip_for_outbound_sockets=127.2.74.193
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:34549
--builtin_ntp_servers=127.2.74.212:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:45:00.487973 21049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:00.488283 21049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:00.488327 21049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:00.492708 21049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:00.492789 21049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:00.492851 21049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:45:00.498015 21049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:34549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:00.499286 21049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:00.500808 21049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:45:00.511148 21049 server_base.cc:1034] running on GCE node
W20260925 22:45:00.511101 21057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:00.511132 21054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:00.511101 21055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:00.511991 21049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:00.512694 21049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:00.513903 21049 hybrid_clock.cc:648] HybridClock initialized: now 1790376300513886 us; error 32 us; skew 500 ppm
I20260925 22:45:00.516153 21049 webserver.cc:460] Webserver started at http://127.2.74.193:40925/ using document root <none> and password file <none>
I20260925 22:45:00.516929 21049 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:00.517038 21049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:00.517316 21049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:00.519814 21049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-0/data/instance:
uuid: "725cb20ad1f142259a7d23cee32cc642"
format_stamp: "Formatted at 2026-09-25 22:45:00 on dist-test-slave-csnl"
server_key: "869e0fd06a178cc1d11d0ba03d45169b"
server_key_iv: "9690b4346ae1e1825477b6a63a9e2222"
server_key_version: "encryptionkey@0"
I20260925 22:45:00.520567 21049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance:
uuid: "725cb20ad1f142259a7d23cee32cc642"
format_stamp: "Formatted at 2026-09-25 22:45:00 on dist-test-slave-csnl"
server_key: "869e0fd06a178cc1d11d0ba03d45169b"
server_key_iv: "9690b4346ae1e1825477b6a63a9e2222"
server_key_version: "encryptionkey@0"
I20260925 22:45:00.525123 21049 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:45:00.528137 21063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:00.529657 21049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 22:45:00.529863 21049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "725cb20ad1f142259a7d23cee32cc642"
format_stamp: "Formatted at 2026-09-25 22:45:00 on dist-test-slave-csnl"
server_key: "869e0fd06a178cc1d11d0ba03d45169b"
server_key_iv: "9690b4346ae1e1825477b6a63a9e2222"
server_key_version: "encryptionkey@0"
I20260925 22:45:00.530042 21049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:00.546736 21049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:00.547614 21049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:00.547835 21049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:00.548460 21049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:00.549479 21049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:00.549564 21049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:00.549640 21049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:00.549695 21049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:00.559470 21049 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:32855
I20260925 22:45:00.559496 21176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:32855 every 8 connection(s)
I20260925 22:45:00.560498 21049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:45:00.566628 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 21049
I20260925 22:45:00.566790 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance
I20260925 22:45:00.567129 2347 external_mini_cluster.cc:1428] Setting key acb425fa403da6ebfb37218a176f3cb1
I20260925 22:45:00.569343 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:0
--local_ip_for_outbound_sockets=127.2.74.194
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:34549
--builtin_ntp_servers=127.2.74.212:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:45:00.572398 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:00.562702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:40861 (local address 127.2.74.254:34549)
0925 22:45:00.562894 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:00.562898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:00.564083 (+ 1185us) server_negotiation.cc:407] Connection header received
0925 22:45:00.565495 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:00.565498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:00.565560 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:00.565640 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:00.566855 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:00.567561 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:00.569286 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:00.569590 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:00.569699 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:00.571723 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:00.571729 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:00.571733 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:00.571755 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:00.571788 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:00.571802 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:00.571888 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:00.572052 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 22:45:00.572164 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260925 22:45:00.573289 21179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:00.563166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34549 (local address 127.2.74.193:40861)
0925 22:45:00.563908 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:00.563956 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:00.565271 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:00.565829 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:00.565837 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:00.565929 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:00.566624 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:00.566647 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:00.567756 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:00.567760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:00.569111 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:00.569127 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:00.570207 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:00.570214 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:00.570493 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:00.571342 (+ 849us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:45:00.571343 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:00.571392 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:45:00.571474 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:00.571905 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:00.571913 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:00.571917 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 22:45:00.572021 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 22:45:00.572289 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":128,"threads_started":1}
I20260925 22:45:00.575034 21177 heartbeater.cc:347] Connected to a master server at 127.2.74.254:34549
I20260925 22:45:00.575331 21177 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:00.575938 21177 heartbeater.cc:511] Master 127.2.74.254:34549 requested a full tablet report, sending...
I20260925 22:45:00.577731 20989 ts_manager.cc:194] Registered new tserver with Master: 725cb20ad1f142259a7d23cee32cc642 (127.2.74.193:32855)
I20260925 22:45:00.579056 20989 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.74.193:40861
W20260925 22:45:00.686862 21180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:00.687186 21180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:00.687227 21180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:00.691838 21180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:00.691918 21180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:00.691982 21180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:45:00.697248 21180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:34549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:00.698418 21180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:00.699901 21180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:00.710325 21188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:00.710328 21186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:00.710372 21185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:00.710507 21180 server_base.cc:1034] running on GCE node
I20260925 22:45:00.711280 21180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:00.712047 21180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:00.713243 21180 hybrid_clock.cc:648] HybridClock initialized: now 1790376300713209 us; error 54 us; skew 500 ppm
I20260925 22:45:00.715440 21180 webserver.cc:460] Webserver started at http://127.2.74.194:35625/ using document root <none> and password file <none>
I20260925 22:45:00.716078 21180 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:00.716167 21180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:00.716427 21180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:00.718755 21180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-1/data/instance:
uuid: "29404f4e853c48d784dd53425e82abec"
format_stamp: "Formatted at 2026-09-25 22:45:00 on dist-test-slave-csnl"
server_key: "2dc4a52ed9e9a75461523f52a6bc2b47"
server_key_iv: "ff578cd58dec31aedc3492378f51a851"
server_key_version: "encryptionkey@0"
I20260925 22:45:00.719519 21180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance:
uuid: "29404f4e853c48d784dd53425e82abec"
format_stamp: "Formatted at 2026-09-25 22:45:00 on dist-test-slave-csnl"
server_key: "2dc4a52ed9e9a75461523f52a6bc2b47"
server_key_iv: "ff578cd58dec31aedc3492378f51a851"
server_key_version: "encryptionkey@0"
I20260925 22:45:00.723974 21180 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:45:00.727052 21194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:00.728677 21180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:45:00.728896 21180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "29404f4e853c48d784dd53425e82abec"
format_stamp: "Formatted at 2026-09-25 22:45:00 on dist-test-slave-csnl"
server_key: "2dc4a52ed9e9a75461523f52a6bc2b47"
server_key_iv: "ff578cd58dec31aedc3492378f51a851"
server_key_version: "encryptionkey@0"
I20260925 22:45:00.729069 21180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:00.743700 21180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:00.744520 21180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:00.744750 21180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:00.745393 21180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:00.746428 21180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:00.746515 21180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:00.746588 21180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:00.746642 21180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:00.756218 21180 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:44071
I20260925 22:45:00.756235 21307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:44071 every 8 connection(s)
I20260925 22:45:00.757222 21180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:45:00.767251 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 21180
I20260925 22:45:00.767316 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:00.759286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:33953 (local address 127.2.74.254:34549)
0925 22:45:00.759452 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:00.759455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:00.760590 (+ 1135us) server_negotiation.cc:407] Connection header received
0925 22:45:00.762103 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:00.762106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:00.762173 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:00.762284 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:00.763376 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:00.763916 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:00.765129 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:00.765362 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:00.765485 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:00.766724 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:00.766728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:00.766731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:00.766748 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:00.766775 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:00.766783 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:00.766864 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:00.767120 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 22:45:00.767188 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 22:45:00.767391 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance
I20260925 22:45:00.767704 2347 external_mini_cluster.cc:1428] Setting key 07ee8f04f3c38d7e4b7815788c96016d
I20260925 22:45:00.768538 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:00.759735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34549 (local address 127.2.74.194:33953)
0925 22:45:00.760418 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:00.760462 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:00.761846 (+ 1384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:00.762481 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:00.762489 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:00.762590 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:00.763219 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:00.763236 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:00.764071 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:00.764075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:00.764984 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:00.764993 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:00.765535 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:00.765539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:00.765723 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:00.766416 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:45:00.766417 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:00.766463 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:45:00.766538 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:00.766879 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:00.766890 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:00.766903 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 22:45:00.767092 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 22:45:00.767448 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":126,"threads_started":1}
I20260925 22:45:00.769878 21308 heartbeater.cc:347] Connected to a master server at 127.2.74.254:34549
I20260925 22:45:00.770161 21308 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:00.770218 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:0
--local_ip_for_outbound_sockets=127.2.74.195
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:34549
--builtin_ntp_servers=127.2.74.212:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:45:00.770843 21308 heartbeater.cc:511] Master 127.2.74.254:34549 requested a full tablet report, sending...
I20260925 22:45:00.772213 20989 ts_manager.cc:194] Registered new tserver with Master: 29404f4e853c48d784dd53425e82abec (127.2.74.194:44071)
I20260925 22:45:00.772934 20989 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.74.194:33953
W20260925 22:45:00.892396 21311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:00.892709 21311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:00.892750 21311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:00.897125 21311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:00.897192 21311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:00.897250 21311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:45:00.902354 21311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:34549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:00.903486 21311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:00.904855 21311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:45:00.915270 21311 server_base.cc:1034] running on GCE node
W20260925 22:45:00.915196 21319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:00.915313 21316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:00.915196 21317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:00.915835 21311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:00.916579 21311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:00.917788 21311 hybrid_clock.cc:648] HybridClock initialized: now 1790376300917742 us; error 52 us; skew 500 ppm
I20260925 22:45:00.920079 21311 webserver.cc:460] Webserver started at http://127.2.74.195:39143/ using document root <none> and password file <none>
I20260925 22:45:00.920858 21311 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:00.920938 21311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:00.921224 21311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:00.923669 21311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-2/data/instance:
uuid: "07323eb639f74d298704ed86d67c717a"
format_stamp: "Formatted at 2026-09-25 22:45:00 on dist-test-slave-csnl"
server_key: "04d6ccfbdb60cb10b64b751be1111204"
server_key_iv: "43fff4c94f21cdbbbecb60910f9339b1"
server_key_version: "encryptionkey@0"
I20260925 22:45:00.924293 21311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance:
uuid: "07323eb639f74d298704ed86d67c717a"
format_stamp: "Formatted at 2026-09-25 22:45:00 on dist-test-slave-csnl"
server_key: "04d6ccfbdb60cb10b64b751be1111204"
server_key_iv: "43fff4c94f21cdbbbecb60910f9339b1"
server_key_version: "encryptionkey@0"
I20260925 22:45:00.928411 21311 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:45:00.931342 21325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:00.932721 21311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:45:00.932901 21311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "07323eb639f74d298704ed86d67c717a"
format_stamp: "Formatted at 2026-09-25 22:45:00 on dist-test-slave-csnl"
server_key: "04d6ccfbdb60cb10b64b751be1111204"
server_key_iv: "43fff4c94f21cdbbbecb60910f9339b1"
server_key_version: "encryptionkey@0"
I20260925 22:45:00.933074 21311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:00.986635 21311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:00.987600 21311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:00.987861 21311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:00.988571 21311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:00.989679 21311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:00.989768 21311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:00.989851 21311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:00.989902 21311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:00.999915 21311 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:36007
I20260925 22:45:00.999939 21438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:36007 every 8 connection(s)
I20260925 22:45:01.001049 21311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:45:01.008670 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 21311
I20260925 22:45:01.008831 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance
I20260925 22:45:01.009142 2347 external_mini_cluster.cc:1428] Setting key 2efce6d1f14ae13a9c615f31cb3b382e
I20260925 22:45:01.011687 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:01.003227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:49211 (local address 127.2.74.254:34549)
0925 22:45:01.003374 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:01.003378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:01.004514 (+ 1136us) server_negotiation.cc:407] Connection header received
0925 22:45:01.006015 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:01.006019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:01.006088 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:01.006200 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:01.007191 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:01.007756 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:01.009189 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:01.009510 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:01.009641 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:01.011139 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:01.011143 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:01.011146 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:01.011159 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:01.011182 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:01.011193 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:01.011276 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:01.011444 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 22:45:01.011538 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:45:01.012753 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:01.003653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34549 (local address 127.2.74.195:49211)
0925 22:45:01.004341 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:01.004385 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:01.005758 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:01.006393 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:01.006400 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:01.006499 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:01.007034 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:01.007048 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:01.007908 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:01.007917 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:01.009043 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:01.009055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:01.009722 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:01.009726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:01.009956 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:01.010775 (+ 819us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:45:01.010777 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:01.010835 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:45:01.010912 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:01.011284 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:01.011291 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:01.011296 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:45:01.011426 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 22:45:01.011782 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":135,"threads_started":1}
I20260925 22:45:01.014108 21439 heartbeater.cc:347] Connected to a master server at 127.2.74.254:34549
I20260925 22:45:01.014420 21439 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:01.015082 21439 heartbeater.cc:511] Master 127.2.74.254:34549 requested a full tablet report, sending...
I20260925 22:45:01.016357 20989 ts_manager.cc:194] Registered new tserver with Master: 07323eb639f74d298704ed86d67c717a (127.2.74.195:36007)
I20260925 22:45:01.016983 20989 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.74.195:49211
I20260925 22:45:01.023897 2347 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:45:01.030117 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:01.026065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37734 (local address 127.2.74.254:34549)
0925 22:45:01.026228 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:01.026232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:01.026303 (+ 71us) server_negotiation.cc:407] Connection header received
0925 22:45:01.026417 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:01.026419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:01.026466 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:01.026539 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:01.026787 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:01.027390 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:01.028288 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:01.028515 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:01.028629 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:01.029562 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:01.029566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:01.029575 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:01.029590 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:01.029610 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:01.029618 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:01.029749 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:01.029895 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 22:45:01.029938 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 22:45:01.033916 20989 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37734:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:45:01.037467 20989 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:45:01.054577 21451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:01.047933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43460 (local address 127.2.74.193:32855)
0925 22:45:01.049029 (+ 1096us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:01.049035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:01.049063 (+ 28us) server_negotiation.cc:407] Connection header received
0925 22:45:01.049150 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:01.049154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:01.049386 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:01.050082 (+ 696us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:01.050236 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:01.051118 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:01.053056 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:01.053802 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:01.053940 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:01.054166 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:01.054276 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 22:45:01.054387 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":996,"thread_start_us":82,"threads_started":1}
I20260925 22:45:01.054580 21450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:01.047751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:32855 (local address 127.0.0.1:43460)
0925 22:45:01.048188 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:01.048233 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:01.048387 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:01.049592 (+ 1205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:01.049598 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:01.049620 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:01.049817 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:01.049827 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:01.051245 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:01.051250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:01.052882 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:01.052896 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:01.053944 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:01.053948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:01.054068 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:01.054112 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 22:45:01.054243 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 22:45:01.054387 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":1,"thread_start_us":132,"threads_started":1}
I20260925 22:45:01.054909 21452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:01.048356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:44071 (local address 127.0.0.1:47346)
0925 22:45:01.049283 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:01.049296 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:01.049413 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:01.049904 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:01.049907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:01.049925 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:01.050059 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:01.050066 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:01.051707 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:01.051711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:01.053155 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:01.053165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:01.054236 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:01.054239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:01.054425 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:01.054448 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:45:01.054524 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 22:45:01.054601 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":54,"thread_start_us":132,"threads_started":1}
I20260925 22:45:01.055020 21454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:01.048556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47346 (local address 127.2.74.194:44071)
0925 22:45:01.049008 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:01.049035 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:01.049394 (+ 359us) server_negotiation.cc:407] Connection header received
0925 22:45:01.049552 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:01.049557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:01.049750 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:01.050192 (+ 442us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:01.050363 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:01.051572 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:01.053290 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:01.054100 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:01.054448 (+ 348us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:01.054647 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:01.054701 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 22:45:01.054818 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":137,"threads_started":1}
I20260925 22:45:01.055645 21453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:01.048566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:36007 (local address 127.0.0.1:34636)
0925 22:45:01.048856 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:01.048870 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:01.048984 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:01.050084 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:01.050087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:01.050103 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:01.050181 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:01.050187 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:01.051737 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:01.051741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:01.052823 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:01.052835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:01.054236 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:01.054239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:01.055317 (+ 1078us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:01.055337 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 22:45:01.055432 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 22:45:01.055520 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":88,"thread_start_us":96,"threads_started":1}
I20260925 22:45:01.056324 21455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:01.048779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34636 (local address 127.2.74.195:36007)
0925 22:45:01.049675 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:01.049686 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:01.049708 (+ 22us) server_negotiation.cc:407] Connection header received
0925 22:45:01.049777 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:01.049781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:01.049938 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:01.050088 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:01.050318 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:01.051573 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:01.052979 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:01.053819 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:01.053939 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:01.054169 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:01.055470 (+ 1301us) server_negotiation.cc:299] Negotiation successful
0925 22:45:01.056159 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":165,"threads_started":1}
I20260925 22:45:01.058130 21111 tablet_service.cc:1467] Processing CreateTablet for tablet 2f41c2ae2cac42b3ac1e1d106b6e911c (DEFAULT_TABLE table=test-table [id=6cd0f4f7228e4cd2963c8334cec0ccea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:01.058130 21242 tablet_service.cc:1467] Processing CreateTablet for tablet 2f41c2ae2cac42b3ac1e1d106b6e911c (DEFAULT_TABLE table=test-table [id=6cd0f4f7228e4cd2963c8334cec0ccea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:01.058780 21373 tablet_service.cc:1467] Processing CreateTablet for tablet 2f41c2ae2cac42b3ac1e1d106b6e911c (DEFAULT_TABLE table=test-table [id=6cd0f4f7228e4cd2963c8334cec0ccea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:01.059433 21242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f41c2ae2cac42b3ac1e1d106b6e911c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:01.059434 21111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f41c2ae2cac42b3ac1e1d106b6e911c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:01.059861 21373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f41c2ae2cac42b3ac1e1d106b6e911c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:01.067149 21456 tablet_bootstrap.cc:492] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 07323eb639f74d298704ed86d67c717a: Bootstrap starting.
I20260925 22:45:01.067791 21457 tablet_bootstrap.cc:492] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 725cb20ad1f142259a7d23cee32cc642: Bootstrap starting.
I20260925 22:45:01.067930 21458 tablet_bootstrap.cc:492] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 29404f4e853c48d784dd53425e82abec: Bootstrap starting.
I20260925 22:45:01.069581 21456 tablet_bootstrap.cc:654] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 07323eb639f74d298704ed86d67c717a: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:01.070036 21458 tablet_bootstrap.cc:654] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 29404f4e853c48d784dd53425e82abec: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:01.070034 21457 tablet_bootstrap.cc:654] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 725cb20ad1f142259a7d23cee32cc642: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:01.070468 21456 log.cc:824] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 07323eb639f74d298704ed86d67c717a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:01.070885 21457 log.cc:824] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 725cb20ad1f142259a7d23cee32cc642: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:01.070884 21458 log.cc:824] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 29404f4e853c48d784dd53425e82abec: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:01.072758 21456 tablet_bootstrap.cc:492] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 07323eb639f74d298704ed86d67c717a: No bootstrap required, opened a new log
I20260925 22:45:01.072992 21456 ts_tablet_manager.cc:1397] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 07323eb639f74d298704ed86d67c717a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:45:01.073187 21458 tablet_bootstrap.cc:492] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 29404f4e853c48d784dd53425e82abec: No bootstrap required, opened a new log
I20260925 22:45:01.073408 21458 ts_tablet_manager.cc:1397] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 29404f4e853c48d784dd53425e82abec: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:45:01.073580 21457 tablet_bootstrap.cc:492] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 725cb20ad1f142259a7d23cee32cc642: No bootstrap required, opened a new log
I20260925 22:45:01.073799 21457 ts_tablet_manager.cc:1397] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 725cb20ad1f142259a7d23cee32cc642: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:45:01.077068 21457 raft_consensus.cc:348] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 725cb20ad1f142259a7d23cee32cc642 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725cb20ad1f142259a7d23cee32cc642" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 32855 } } peers { permanent_uuid: "07323eb639f74d298704ed86d67c717a" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36007 } } peers { permanent_uuid: "29404f4e853c48d784dd53425e82abec" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44071 } }
I20260925 22:45:01.077349 21457 raft_consensus.cc:374] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 725cb20ad1f142259a7d23cee32cc642 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:01.077404 21457 raft_consensus.cc:729] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 725cb20ad1f142259a7d23cee32cc642 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 725cb20ad1f142259a7d23cee32cc642, State: Initialized, Role: FOLLOWER
I20260925 22:45:01.077606 21456 raft_consensus.cc:348] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 07323eb639f74d298704ed86d67c717a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725cb20ad1f142259a7d23cee32cc642" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 32855 } } peers { permanent_uuid: "07323eb639f74d298704ed86d67c717a" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36007 } } peers { permanent_uuid: "29404f4e853c48d784dd53425e82abec" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44071 } }
I20260925 22:45:01.077606 21458 raft_consensus.cc:348] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 29404f4e853c48d784dd53425e82abec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725cb20ad1f142259a7d23cee32cc642" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 32855 } } peers { permanent_uuid: "07323eb639f74d298704ed86d67c717a" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36007 } } peers { permanent_uuid: "29404f4e853c48d784dd53425e82abec" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44071 } }
I20260925 22:45:01.077867 21456 raft_consensus.cc:374] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 07323eb639f74d298704ed86d67c717a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:01.077867 21458 raft_consensus.cc:374] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 29404f4e853c48d784dd53425e82abec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:01.077935 21458 raft_consensus.cc:729] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 29404f4e853c48d784dd53425e82abec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29404f4e853c48d784dd53425e82abec, State: Initialized, Role: FOLLOWER
I20260925 22:45:01.077927 21456 raft_consensus.cc:729] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 07323eb639f74d298704ed86d67c717a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07323eb639f74d298704ed86d67c717a, State: Initialized, Role: FOLLOWER
I20260925 22:45:01.077947 21457 consensus_queue.cc:260] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 725cb20ad1f142259a7d23cee32cc642 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725cb20ad1f142259a7d23cee32cc642" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 32855 } } peers { permanent_uuid: "07323eb639f74d298704ed86d67c717a" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36007 } } peers { permanent_uuid: "29404f4e853c48d784dd53425e82abec" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44071 } }
I20260925 22:45:01.078408 21456 consensus_queue.cc:260] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 07323eb639f74d298704ed86d67c717a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725cb20ad1f142259a7d23cee32cc642" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 32855 } } peers { permanent_uuid: "07323eb639f74d298704ed86d67c717a" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36007 } } peers { permanent_uuid: "29404f4e853c48d784dd53425e82abec" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44071 } }
I20260925 22:45:01.078689 21177 heartbeater.cc:503] Master 127.2.74.254:34549 was elected leader, sending a full tablet report...
I20260925 22:45:01.079180 21457 ts_tablet_manager.cc:1428] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 725cb20ad1f142259a7d23cee32cc642: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260925 22:45:01.079429 21456 ts_tablet_manager.cc:1428] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 07323eb639f74d298704ed86d67c717a: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260925 22:45:01.078408 21458 consensus_queue.cc:260] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 29404f4e853c48d784dd53425e82abec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725cb20ad1f142259a7d23cee32cc642" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 32855 } } peers { permanent_uuid: "07323eb639f74d298704ed86d67c717a" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36007 } } peers { permanent_uuid: "29404f4e853c48d784dd53425e82abec" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44071 } }
I20260925 22:45:01.079756 21439 heartbeater.cc:503] Master 127.2.74.254:34549 was elected leader, sending a full tablet report...
I20260925 22:45:01.080379 21308 heartbeater.cc:503] Master 127.2.74.254:34549 was elected leader, sending a full tablet report...
I20260925 22:45:01.080582 21458 ts_tablet_manager.cc:1428] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 29404f4e853c48d784dd53425e82abec: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 22:45:01.177561 21462 raft_consensus.cc:482] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 725cb20ad1f142259a7d23cee32cc642 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:45:01.177786 21462 raft_consensus.cc:504] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 725cb20ad1f142259a7d23cee32cc642 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725cb20ad1f142259a7d23cee32cc642" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 32855 } } peers { permanent_uuid: "07323eb639f74d298704ed86d67c717a" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36007 } } peers { permanent_uuid: "29404f4e853c48d784dd53425e82abec" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44071 } }
I20260925 22:45:01.179059 21462 leader_election.cc:288] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 725cb20ad1f142259a7d23cee32cc642 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07323eb639f74d298704ed86d67c717a (127.2.74.195:36007), 29404f4e853c48d784dd53425e82abec (127.2.74.194:44071)
I20260925 22:45:01.183238 21455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:01.179383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:59671 (local address 127.2.74.195:36007)
0925 22:45:01.179517 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:01.179522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:01.179791 (+ 269us) server_negotiation.cc:407] Connection header received
0925 22:45:01.179967 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:01.179971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:01.180011 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:01.180109 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:01.180417 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:01.180930 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:01.182130 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:01.182538 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:01.182685 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:01.182737 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:01.183002 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 22:45:01.183089 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:45:01.183238 21454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:01.179430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:45097 (local address 127.2.74.194:44071)
0925 22:45:01.179559 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:01.179578 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:01.179869 (+ 291us) server_negotiation.cc:407] Connection header received
0925 22:45:01.180024 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:01.180028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:01.180080 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:01.180167 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:01.180434 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:01.180950 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:01.182128 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:01.182534 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:01.182667 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:01.182720 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:01.183010 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 22:45:01.183096 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:45:01.183264 21465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:01.179281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:36007 (local address 127.2.74.193:59671)
0925 22:45:01.179678 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:01.179690 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:01.179835 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:01.180122 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:01.180125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:01.180141 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:01.180295 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:01.180300 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:01.181052 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:01.181057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:01.181984 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:01.181994 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:01.182675 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:01.182679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:01.182839 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:01.182867 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 22:45:01.182960 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 22:45:01.183098 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":86,"thread_start_us":127,"threads_started":1}
I20260925 22:45:01.183264 21466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:01.179290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:44071 (local address 127.2.74.193:45097)
0925 22:45:01.179761 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:01.179772 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:01.179902 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:01.180179 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:01.180182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:01.180198 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:01.180318 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:01.180324 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:01.181070 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:01.181076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:01.181983 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:01.181993 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:01.182666 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:01.182670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:01.182862 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:01.182880 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 22:45:01.182990 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 22:45:01.183108 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":118,"thread_start_us":146,"threads_started":1}
I20260925 22:45:01.184044 21393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f41c2ae2cac42b3ac1e1d106b6e911c" candidate_uuid: "725cb20ad1f142259a7d23cee32cc642" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07323eb639f74d298704ed86d67c717a" is_pre_election: true
I20260925 22:45:01.184113 21262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f41c2ae2cac42b3ac1e1d106b6e911c" candidate_uuid: "725cb20ad1f142259a7d23cee32cc642" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29404f4e853c48d784dd53425e82abec" is_pre_election: true
I20260925 22:45:01.184425 21262 raft_consensus.cc:2454] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 29404f4e853c48d784dd53425e82abec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 725cb20ad1f142259a7d23cee32cc642 in term 0.
I20260925 22:45:01.184425 21393 raft_consensus.cc:2454] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 07323eb639f74d298704ed86d67c717a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 725cb20ad1f142259a7d23cee32cc642 in term 0.
I20260925 22:45:01.184988 21067 leader_election.cc:302] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 725cb20ad1f142259a7d23cee32cc642 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 07323eb639f74d298704ed86d67c717a, 29404f4e853c48d784dd53425e82abec, 725cb20ad1f142259a7d23cee32cc642; no voters:
I20260925 22:45:01.185307 21462 raft_consensus.cc:2781] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 725cb20ad1f142259a7d23cee32cc642 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:45:01.185395 21462 raft_consensus.cc:482] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 725cb20ad1f142259a7d23cee32cc642 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:45:01.185437 21462 raft_consensus.cc:3037] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 725cb20ad1f142259a7d23cee32cc642 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:01.186502 21462 raft_consensus.cc:504] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 725cb20ad1f142259a7d23cee32cc642 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725cb20ad1f142259a7d23cee32cc642" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 32855 } } peers { permanent_uuid: "07323eb639f74d298704ed86d67c717a" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36007 } } peers { permanent_uuid: "29404f4e853c48d784dd53425e82abec" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44071 } }
I20260925 22:45:01.186882 21462 leader_election.cc:288] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 725cb20ad1f142259a7d23cee32cc642 [CANDIDATE]: Term 1 election: Requested vote from peers 07323eb639f74d298704ed86d67c717a (127.2.74.195:36007), 29404f4e853c48d784dd53425e82abec (127.2.74.194:44071)
I20260925 22:45:01.187268 21393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f41c2ae2cac42b3ac1e1d106b6e911c" candidate_uuid: "725cb20ad1f142259a7d23cee32cc642" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07323eb639f74d298704ed86d67c717a"
I20260925 22:45:01.187322 21262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f41c2ae2cac42b3ac1e1d106b6e911c" candidate_uuid: "725cb20ad1f142259a7d23cee32cc642" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29404f4e853c48d784dd53425e82abec"
I20260925 22:45:01.187417 21393 raft_consensus.cc:3037] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 07323eb639f74d298704ed86d67c717a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:01.187457 21262 raft_consensus.cc:3037] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 29404f4e853c48d784dd53425e82abec [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:01.188795 21393 raft_consensus.cc:2454] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 07323eb639f74d298704ed86d67c717a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 725cb20ad1f142259a7d23cee32cc642 in term 1.
I20260925 22:45:01.188795 21262 raft_consensus.cc:2454] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 29404f4e853c48d784dd53425e82abec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 725cb20ad1f142259a7d23cee32cc642 in term 1.
I20260925 22:45:01.189126 21067 leader_election.cc:302] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 725cb20ad1f142259a7d23cee32cc642 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07323eb639f74d298704ed86d67c717a, 725cb20ad1f142259a7d23cee32cc642; no voters:
I20260925 22:45:01.189311 21462 raft_consensus.cc:2781] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 725cb20ad1f142259a7d23cee32cc642 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:01.189563 21462 raft_consensus.cc:686] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 725cb20ad1f142259a7d23cee32cc642 [term 1 LEADER]: Becoming Leader. State: Replica: 725cb20ad1f142259a7d23cee32cc642, State: Running, Role: LEADER
I20260925 22:45:01.189957 21462 consensus_queue.cc:237] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 725cb20ad1f142259a7d23cee32cc642 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725cb20ad1f142259a7d23cee32cc642" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 32855 } } peers { permanent_uuid: "07323eb639f74d298704ed86d67c717a" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36007 } } peers { permanent_uuid: "29404f4e853c48d784dd53425e82abec" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44071 } }
I20260925 22:45:01.193344 20989 catalog_manager.cc:5495] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 725cb20ad1f142259a7d23cee32cc642 reported cstate change: term changed from 0 to 1, leader changed from <none> to 725cb20ad1f142259a7d23cee32cc642 (127.2.74.193). New cstate: current_term: 1 leader_uuid: "725cb20ad1f142259a7d23cee32cc642" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725cb20ad1f142259a7d23cee32cc642" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 32855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07323eb639f74d298704ed86d67c717a" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29404f4e853c48d784dd53425e82abec" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44071 } health_report { overall_health: UNKNOWN } } }
I20260925 22:45:01.210843 21451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:01.207071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43466 (local address 127.2.74.193:32855)
0925 22:45:01.207233 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:01.207238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:01.207252 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:45:01.207399 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:01.207401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:01.207460 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:01.207599 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:45:01.207853 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:01.208374 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:01.209278 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:01.209501 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:01.209624 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:01.209963 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:45:01.210400 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:45:01.210577 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:01.210638 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 22:45:01.210677 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 22:45:01.217726 21393 raft_consensus.cc:1261] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 07323eb639f74d298704ed86d67c717a [term 1 FOLLOWER]: Refusing update from remote peer 725cb20ad1f142259a7d23cee32cc642: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:01.217727 21262 raft_consensus.cc:1261] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 29404f4e853c48d784dd53425e82abec [term 1 FOLLOWER]: Refusing update from remote peer 725cb20ad1f142259a7d23cee32cc642: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:01.218717 21467 consensus_queue.cc:1035] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 725cb20ad1f142259a7d23cee32cc642 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07323eb639f74d298704ed86d67c717a" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 36007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:01.218981 21462 consensus_queue.cc:1035] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 725cb20ad1f142259a7d23cee32cc642 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29404f4e853c48d784dd53425e82abec" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 44071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:01.227842 21470 mvcc.cc:205] Tried to move back new op lower bound from 7333381329779253248 to 7333381329677193216. Current Snapshot: MvccSnapshot[applied={T|T < 7333381329779253248}]
I20260925 22:45:01.229187 21471 mvcc.cc:205] Tried to move back new op lower bound from 7333381329779253248 to 7333381329677193216. Current Snapshot: MvccSnapshot[applied={T|T < 7333381329779253248}]
I20260925 22:45:01.229712 21472 mvcc.cc:205] Tried to move back new op lower bound from 7333381329779253248 to 7333381329677193216. Current Snapshot: MvccSnapshot[applied={T|T < 7333381329779253248}]
I20260925 22:45:01.238292 20988 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:45:01.238497 20988 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:45:01.241175 20988 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d82ad1dab474037a040936bf535dfc0: Sending DeleteTablet for 3 replicas of tablet 2f41c2ae2cac42b3ac1e1d106b6e911c
I20260925 22:45:01.242106 21373 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f41c2ae2cac42b3ac1e1d106b6e911c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:01 UTC) from {username='slave'} at 127.0.0.1:34636
I20260925 22:45:01.242105 21111 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f41c2ae2cac42b3ac1e1d106b6e911c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:01 UTC) from {username='slave'} at 127.0.0.1:43460
I20260925 22:45:01.242730 21480 tablet_replica.cc:330] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 07323eb639f74d298704ed86d67c717a: stopping tablet replica
I20260925 22:45:01.242726 21242 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f41c2ae2cac42b3ac1e1d106b6e911c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:01 UTC) from {username='slave'} at 127.0.0.1:47346
I20260925 22:45:01.243135 21479 tablet_replica.cc:330] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 725cb20ad1f142259a7d23cee32cc642: stopping tablet replica
I20260925 22:45:01.243268 21481 tablet_replica.cc:330] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 29404f4e853c48d784dd53425e82abec: stopping tablet replica
I20260925 22:45:01.243678 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 21049
I20260925 22:45:01.243661 21481 raft_consensus.cc:2229] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 29404f4e853c48d784dd53425e82abec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:45:01.244009 21479 raft_consensus.cc:2229] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 725cb20ad1f142259a7d23cee32cc642 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:45:01.244197 21480 raft_consensus.cc:2229] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 07323eb639f74d298704ed86d67c717a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:45:01.245127 21481 raft_consensus.cc:2258] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 29404f4e853c48d784dd53425e82abec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:45:01.246577 21480 raft_consensus.cc:2258] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 07323eb639f74d298704ed86d67c717a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:45:01.247257 21481 ts_tablet_manager.cc:1905] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 29404f4e853c48d784dd53425e82abec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:45:01.248430 21480 ts_tablet_manager.cc:1905] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 07323eb639f74d298704ed86d67c717a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:45:01.249760 21481 ts_tablet_manager.cc:1918] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 29404f4e853c48d784dd53425e82abec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:45:01.249897 21481 log.cc:1196] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 29404f4e853c48d784dd53425e82abec: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-1/wal/wals/2f41c2ae2cac42b3ac1e1d106b6e911c
I20260925 22:45:01.250223 21481 ts_tablet_manager.cc:1939] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 29404f4e853c48d784dd53425e82abec: Deleting consensus metadata
I20260925 22:45:01.251716 20973 catalog_manager.cc:4841] TS 29404f4e853c48d784dd53425e82abec (127.2.74.194:44071): tablet 2f41c2ae2cac42b3ac1e1d106b6e911c (table test-table [id=6cd0f4f7228e4cd2963c8334cec0ccea]) successfully deleted
I20260925 22:45:01.252146 21480 ts_tablet_manager.cc:1918] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 07323eb639f74d298704ed86d67c717a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:45:01.252275 21480 log.cc:1196] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 07323eb639f74d298704ed86d67c717a: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376047580784-2347-0/minicluster-data/ts-2/wal/wals/2f41c2ae2cac42b3ac1e1d106b6e911c
I20260925 22:45:01.252635 21480 ts_tablet_manager.cc:1939] T 2f41c2ae2cac42b3ac1e1d106b6e911c P 07323eb639f74d298704ed86d67c717a: Deleting consensus metadata
W20260925 22:45:01.253083 20976 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.74.193:32855 (error 108)
I20260925 22:45:01.253971 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 21180
I20260925 22:45:01.254194 20976 catalog_manager.cc:4841] TS 07323eb639f74d298704ed86d67c717a (127.2.74.195:36007): tablet 2f41c2ae2cac42b3ac1e1d106b6e911c (table test-table [id=6cd0f4f7228e4cd2963c8334cec0ccea]) successfully deleted
I20260925 22:45:01.254730 21453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:01.254018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:32855 (local address 127.0.0.1:43476)
0925 22:45:01.254556 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:01.254634 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.193:32855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436}
W20260925 22:45:01.255174 20976 catalog_manager.cc:4584] TS 725cb20ad1f142259a7d23cee32cc642 (127.2.74.193:32855): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f41c2ae2cac42b3ac1e1d106b6e911c: Network error: Client connection negotiation failed: client connection to 127.2.74.193:32855: connect: Connection refused (error 111)
I20260925 22:45:01.260761 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 21311
I20260925 22:45:01.267362 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 20958
2026-09-25T22:45:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1202 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:01 dist-test-slave-csnl krb5kdc[21485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:01 dist-test-slave-csnl krb5kdc[21485](info): set up 2 sockets
Sep 25 22:45:01 dist-test-slave-csnl krb5kdc[21485](info): commencing operation
krb5kdc: starting...
W20260925 22:45:02.303155 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:45:02 dist-test-slave-csnl krb5kdc[21485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:45:02.434468 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:33957
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:33957
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:02.548205 21498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:02.548569 21498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:02.548653 21498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:02.552955 21498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:02.553085 21498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:02.553124 21498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:02.553165 21498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:02.553198 21498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:02.558925 21498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:33957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:33957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:02.560242 21498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:02.561352 21498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:02.571337 21506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:02.571337 21504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:02.571530 21503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:02.571717 21498 server_base.cc:1034] running on GCE node
I20260925 22:45:02.572929 21498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:02.574823 21498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:02.576151 21498 hybrid_clock.cc:648] HybridClock initialized: now 1790376302576100 us; error 75 us; skew 500 ppm
Sep 25 22:45:02 dist-test-slave-csnl krb5kdc[21485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376302, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:02.579617 21498 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:45:02.580883 21498 webserver.cc:460] Webserver started at http://127.2.74.254:38973/ using document root <none> and password file <none>
I20260925 22:45:02.581511 21498 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:02.581580 21498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:02.581777 21498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:02.584260 21498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "2f66bb5c9e5b40aa9c97c2f0c3291146"
format_stamp: "Formatted at 2026-09-25 22:45:02 on dist-test-slave-csnl"
server_key: "16e521556b599a4c6a0655a4979a1f76"
server_key_iv: "09bd659b006537462fe4256672fb68a3"
server_key_version: "encryptionkey@0"
I20260925 22:45:02.584870 21498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "2f66bb5c9e5b40aa9c97c2f0c3291146"
format_stamp: "Formatted at 2026-09-25 22:45:02 on dist-test-slave-csnl"
server_key: "16e521556b599a4c6a0655a4979a1f76"
server_key_iv: "09bd659b006537462fe4256672fb68a3"
server_key_version: "encryptionkey@0"
I20260925 22:45:02.589208 21498 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:45:02.592025 21513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:02.593441 21498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:02.593629 21498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "2f66bb5c9e5b40aa9c97c2f0c3291146"
format_stamp: "Formatted at 2026-09-25 22:45:02 on dist-test-slave-csnl"
server_key: "16e521556b599a4c6a0655a4979a1f76"
server_key_iv: "09bd659b006537462fe4256672fb68a3"
server_key_version: "encryptionkey@0"
I20260925 22:45:02.593770 21498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:02.634672 21498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:02.638381 21498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:02.638621 21498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:02.646121 21498 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:33957
I20260925 22:45:02.646150 21565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:33957 every 8 connection(s)
I20260925 22:45:02.647166 21498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:02.650010 21566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:02.652055 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 21498
I20260925 22:45:02.652226 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:45:02.652518 2347 external_mini_cluster.cc:1428] Setting key 3ccf0b7f4173b066402c7f8ebdb0355c
I20260925 22:45:02.657198 21566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f66bb5c9e5b40aa9c97c2f0c3291146: Bootstrap starting.
I20260925 22:45:02.660166 21566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f66bb5c9e5b40aa9c97c2f0c3291146: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:02.661190 21566 log.cc:824] T 00000000000000000000000000000000 P 2f66bb5c9e5b40aa9c97c2f0c3291146: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:45:02 dist-test-slave-csnl krb5kdc[21485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:45:02.663856 21566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f66bb5c9e5b40aa9c97c2f0c3291146: No bootstrap required, opened a new log
I20260925 22:45:02.668385 21569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:02.653942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35580 (local address 127.2.74.254:33957)
0925 22:45:02.654382 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:02.654398 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:02.654434 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:45:02.655217 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:02.655239 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:02.655600 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:02.655897 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:02.656849 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:02.657924 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:02.659213 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:02.659712 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:02.659836 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:02.662893 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:02.662902 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:02.662920 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:02.662999 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:02.665047 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:02.665528 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:02.665534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:02.665541 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:02.665684 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:02.665949 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:02.665953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:02.665954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:02.666580 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:02.666793 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:02.667028 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 22:45:02.667349 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":128,"threads_started":1}
I20260925 22:45:02.668736 21566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f66bb5c9e5b40aa9c97c2f0c3291146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f66bb5c9e5b40aa9c97c2f0c3291146" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33957 } }
I20260925 22:45:02.669014 21566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f66bb5c9e5b40aa9c97c2f0c3291146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:02.669072 21566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f66bb5c9e5b40aa9c97c2f0c3291146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f66bb5c9e5b40aa9c97c2f0c3291146, State: Initialized, Role: FOLLOWER
I20260925 22:45:02.669598 21566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f66bb5c9e5b40aa9c97c2f0c3291146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f66bb5c9e5b40aa9c97c2f0c3291146" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33957 } }
I20260925 22:45:02.669759 21566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f66bb5c9e5b40aa9c97c2f0c3291146 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:02.669826 21566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f66bb5c9e5b40aa9c97c2f0c3291146 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:02.669924 21566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f66bb5c9e5b40aa9c97c2f0c3291146 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:02.671159 21566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f66bb5c9e5b40aa9c97c2f0c3291146 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f66bb5c9e5b40aa9c97c2f0c3291146" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33957 } }
I20260925 22:45:02.671468 21566 leader_election.cc:302] T 00000000000000000000000000000000 P 2f66bb5c9e5b40aa9c97c2f0c3291146 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f66bb5c9e5b40aa9c97c2f0c3291146; no voters:
I20260925 22:45:02.671731 21566 leader_election.cc:288] T 00000000000000000000000000000000 P 2f66bb5c9e5b40aa9c97c2f0c3291146 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:02.671900 21571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f66bb5c9e5b40aa9c97c2f0c3291146 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:02.672189 21571 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f66bb5c9e5b40aa9c97c2f0c3291146 [term 1 LEADER]: Becoming Leader. State: Replica: 2f66bb5c9e5b40aa9c97c2f0c3291146, State: Running, Role: LEADER
I20260925 22:45:02.672499 21571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f66bb5c9e5b40aa9c97c2f0c3291146 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f66bb5c9e5b40aa9c97c2f0c3291146" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33957 } }
I20260925 22:45:02.673023 21566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f66bb5c9e5b40aa9c97c2f0c3291146 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:02.674288 21573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f66bb5c9e5b40aa9c97c2f0c3291146 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f66bb5c9e5b40aa9c97c2f0c3291146. Latest consensus state: current_term: 1 leader_uuid: "2f66bb5c9e5b40aa9c97c2f0c3291146" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f66bb5c9e5b40aa9c97c2f0c3291146" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33957 } } }
I20260925 22:45:02.674309 21572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f66bb5c9e5b40aa9c97c2f0c3291146 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f66bb5c9e5b40aa9c97c2f0c3291146" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f66bb5c9e5b40aa9c97c2f0c3291146" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33957 } } }
I20260925 22:45:02.674455 21573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f66bb5c9e5b40aa9c97c2f0c3291146 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:02.674455 21572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f66bb5c9e5b40aa9c97c2f0c3291146 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:02.675050 21581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:02.678138 21581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:02.683718 21581 catalog_manager.cc:1347] Generated new cluster ID: 4b23dbd52ec5442eb88de51b6bc3ad69
I20260925 22:45:02.683801 21581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:02.694149 21581 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:02.695576 21581 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:02.705346 21581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f66bb5c9e5b40aa9c97c2f0c3291146: Generated new TSK 0
I20260925 22:45:02.706315 21581 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
Entry for principal kudu/127.2.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab.
I20260925 22:45:02.738109 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:0
--local_ip_for_outbound_sockets=127.2.74.193
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:33957
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:02.864491 21592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:02.864961 21592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:02.865077 21592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:02.870064 21592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:02.870158 21592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:02.870223 21592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:45:02.876427 21592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.193.keytab
--principal=kudu/127.2.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:33957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:02.877566 21592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:02.878623 21592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:02.889307 21600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:02.889470 21592 server_base.cc:1034] running on GCE node
W20260925 22:45:02.889310 21598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:02.889690 21597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:02.890218 21592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:02.891157 21592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:02.892395 21592 hybrid_clock.cc:648] HybridClock initialized: now 1790376302892375 us; error 39 us; skew 500 ppm
Sep 25 22:45:02 dist-test-slave-csnl krb5kdc[21485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376302, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:02.895494 21592 init.cc:377] Logged in from keytab as kudu/127.2.74.193@KRBTEST.COM (short username kudu)
I20260925 22:45:02.896509 21592 webserver.cc:460] Webserver started at http://127.2.74.193:34319/ using document root <none> and password file <none>
I20260925 22:45:02.897229 21592 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:02.897336 21592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:02.897586 21592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:02.900039 21592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-0/data/instance:
uuid: "1d2fd74a25b2404494d846785ef834bf"
format_stamp: "Formatted at 2026-09-25 22:45:02 on dist-test-slave-csnl"
server_key: "be4f1ac7011b79ac68173b30cf222dfa"
server_key_iv: "5a6b074360766685cb86c0c076b0e68a"
server_key_version: "encryptionkey@0"
I20260925 22:45:02.900676 21592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance:
uuid: "1d2fd74a25b2404494d846785ef834bf"
format_stamp: "Formatted at 2026-09-25 22:45:02 on dist-test-slave-csnl"
server_key: "be4f1ac7011b79ac68173b30cf222dfa"
server_key_iv: "5a6b074360766685cb86c0c076b0e68a"
server_key_version: "encryptionkey@0"
I20260925 22:45:02.904887 21592 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:45:02.908028 21607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:02.909451 21592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:02.909619 21592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "1d2fd74a25b2404494d846785ef834bf"
format_stamp: "Formatted at 2026-09-25 22:45:02 on dist-test-slave-csnl"
server_key: "be4f1ac7011b79ac68173b30cf222dfa"
server_key_iv: "5a6b074360766685cb86c0c076b0e68a"
server_key_version: "encryptionkey@0"
I20260925 22:45:02.909768 21592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:02.929387 21592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:02.932969 21592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:02.933218 21592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:02.933889 21592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:02.934803 21592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:02.934852 21592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:02.934929 21592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:02.935003 21592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260925 22:45:02.944813 21592 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:43181
I20260925 22:45:02.944834 21720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:43181 every 8 connection(s)
I20260925 22:45:02.945816 21592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:45:02.955737 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 21592
Sep 25 22:45:02 dist-test-slave-csnl krb5kdc[21485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376302, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.193@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:45:02.955972 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance
I20260925 22:45:02.956341 2347 external_mini_cluster.cc:1428] Setting key 946530ed2b315386423d111ae50807d0
I20260925 22:45:02.961539 21569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:02.947806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:35105 (local address 127.2.74.254:33957)
0925 22:45:02.947950 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:02.947954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:02.948863 (+ 909us) server_negotiation.cc:407] Connection header received
0925 22:45:02.950172 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:02.950176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:02.950228 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:02.950313 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:02.951715 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:02.952259 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:02.953447 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:02.953686 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:02.953777 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:02.956775 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:02.956779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:02.956782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:02.956810 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:02.958606 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:02.959494 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:02.959498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:02.959499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:02.959590 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:02.960140 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:02.960145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:02.960149 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:02.960435 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:02.960550 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:02.961168 (+ 618us) server_negotiation.cc:299] Negotiation successful
0925 22:45:02.961330 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:45:02.962373 21723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:02.948151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33957 (local address 127.2.74.193:35105)
0925 22:45:02.948685 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:02.948731 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:02.949921 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:02.950552 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:02.950560 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:02.950992 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:02.951546 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:02.951560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:02.952405 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:02.952408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:02.953314 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:02.953324 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:02.953865 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:02.953868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:02.954030 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:02.954735 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:02.954755 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:02.956544 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:02.958760 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:02.958772 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:02.958805 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:02.959333 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:02.959709 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:02.959716 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:02.959721 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:02.959979 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:02.960575 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:02.960582 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:02.960880 (+ 298us) client_negotiation.cc:769] Sending connection context
0925 22:45:02.961057 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 22:45:02.961357 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":139,"threads_started":1}
I20260925 22:45:02.963814 21721 heartbeater.cc:347] Connected to a master server at 127.2.74.254:33957
I20260925 22:45:02.964143 21721 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:02.964843 21721 heartbeater.cc:511] Master 127.2.74.254:33957 requested a full tablet report, sending...
I20260925 22:45:02.966574 21530 ts_manager.cc:194] Registered new tserver with Master: 1d2fd74a25b2404494d846785ef834bf (127.2.74.193:43181)
WARNING: no policy specified for kudu/127.2.74.194@KRBTEST.COM; defaulting to no policy
I20260925 22:45:02.967895 21530 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.193@KRBTEST.COM'} at 127.2.74.193:35105
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
Entry for principal kudu/127.2.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab.
I20260925 22:45:02.988124 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:0
--local_ip_for_outbound_sockets=127.2.74.194
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:33957
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:03.107689 21726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:03.107983 21726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:03.108029 21726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:03.112315 21726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:03.112393 21726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:03.112491 21726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:45:03.118544 21726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.194.keytab
--principal=kudu/127.2.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:33957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:03.119828 21726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:03.120985 21726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:03.132833 21732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:03.132870 21734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:03.132835 21731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:03.133102 21726 server_base.cc:1034] running on GCE node
I20260925 22:45:03.133859 21726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:03.134660 21726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:03.135886 21726 hybrid_clock.cc:648] HybridClock initialized: now 1790376303135855 us; error 56 us; skew 500 ppm
Sep 25 22:45:03 dist-test-slave-csnl krb5kdc[21485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376303, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:03.139032 21726 init.cc:377] Logged in from keytab as kudu/127.2.74.194@KRBTEST.COM (short username kudu)
I20260925 22:45:03.139982 21726 webserver.cc:460] Webserver started at http://127.2.74.194:41287/ using document root <none> and password file <none>
I20260925 22:45:03.140587 21726 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:03.140702 21726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:03.140996 21726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:03.143272 21726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-1/data/instance:
uuid: "1c66c36f472e471083edd0daffc9a5b4"
format_stamp: "Formatted at 2026-09-25 22:45:03 on dist-test-slave-csnl"
server_key: "82b6607d22628c5f9b8869939efe3459"
server_key_iv: "55588846baf710e3346afecbae999a67"
server_key_version: "encryptionkey@0"
I20260925 22:45:03.143877 21726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance:
uuid: "1c66c36f472e471083edd0daffc9a5b4"
format_stamp: "Formatted at 2026-09-25 22:45:03 on dist-test-slave-csnl"
server_key: "82b6607d22628c5f9b8869939efe3459"
server_key_iv: "55588846baf710e3346afecbae999a67"
server_key_version: "encryptionkey@0"
I20260925 22:45:03.148259 21726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 22:45:03.151110 21741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:03.152576 21726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:45:03.152760 21726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "1c66c36f472e471083edd0daffc9a5b4"
format_stamp: "Formatted at 2026-09-25 22:45:03 on dist-test-slave-csnl"
server_key: "82b6607d22628c5f9b8869939efe3459"
server_key_iv: "55588846baf710e3346afecbae999a67"
server_key_version: "encryptionkey@0"
I20260925 22:45:03.152905 21726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:03.173151 21726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:03.176815 21726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:03.177060 21726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:03.177717 21726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:03.178671 21726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:03.178753 21726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:03.178824 21726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:03.178874 21726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:03.188439 21726 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:41769
I20260925 22:45:03.188459 21854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:41769 every 8 connection(s)
I20260925 22:45:03.189467 21726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:45:03.195856 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 21726
I20260925 22:45:03.195986 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance
I20260925 22:45:03.196301 2347 external_mini_cluster.cc:1428] Setting key a89c4a570848a675b1a243b9b4d41e73
Sep 25 22:45:03 dist-test-slave-csnl krb5kdc[21485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376303, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.194@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.74.195@KRBTEST.COM; defaulting to no policy
I20260925 22:45:03.205878 21569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:03.192156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:38853 (local address 127.2.74.254:33957)
0925 22:45:03.192311 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:03.192315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:03.193271 (+ 956us) server_negotiation.cc:407] Connection header received
0925 22:45:03.194687 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:03.194690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:03.194754 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:03.194871 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:03.196319 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.197061 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.198410 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.198663 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.198777 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:03.201773 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:03.201776 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:03.201779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:03.201807 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:03.203455 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:03.204101 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:03.204105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:03.204107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:03.204194 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:03.204573 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:03.204577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:03.204579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:03.204786 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:03.204894 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:03.205577 (+ 683us) server_negotiation.cc:299] Negotiation successful
0925 22:45:03.205710 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 22:45:03.206769 21857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:03.192599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33957 (local address 127.2.74.194:38853)
0925 22:45:03.193107 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:03.193153 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:03.194424 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:03.195103 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:03.195111 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:03.195505 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:03.196149 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:03.196168 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.197203 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.197207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:03.198270 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:03.198282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.198843 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.198848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:03.199098 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:03.199800 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:03.199820 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:03.201542 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:03.203637 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:03.203644 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:03.203664 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:03.203967 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:03.204294 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:03.204297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:03.204299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:03.204459 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:03.204908 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:03.204916 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:03.205318 (+ 402us) client_negotiation.cc:769] Sending connection context
0925 22:45:03.205523 (+ 205us) client_negotiation.cc:240] Negotiation successful
0925 22:45:03.205886 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":146,"threads_started":1}
I20260925 22:45:03.208003 21855 heartbeater.cc:347] Connected to a master server at 127.2.74.254:33957
I20260925 22:45:03.208292 21855 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.195@KRBTEST.COM" created.
I20260925 22:45:03.208847 21855 heartbeater.cc:511] Master 127.2.74.254:33957 requested a full tablet report, sending...
I20260925 22:45:03.210060 21530 ts_manager.cc:194] Registered new tserver with Master: 1c66c36f472e471083edd0daffc9a5b4 (127.2.74.194:41769)
I20260925 22:45:03.210670 21530 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.194@KRBTEST.COM'} at 127.2.74.194:38853
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
Entry for principal kudu/127.2.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab.
I20260925 22:45:03.226325 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:0
--local_ip_for_outbound_sockets=127.2.74.195
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:33957
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:03.343428 21860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:03.343740 21860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:03.343788 21860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:03.348510 21860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:03.348635 21860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:03.348702 21860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:45:03.354678 21860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.195.keytab
--principal=kudu/127.2.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:33957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:03.355952 21860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:03.357098 21860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:03.367915 21868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:03.367903 21865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:03.367903 21866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:03.368183 21860 server_base.cc:1034] running on GCE node
I20260925 22:45:03.368752 21860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:03.369506 21860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:03.370741 21860 hybrid_clock.cc:648] HybridClock initialized: now 1790376303370701 us; error 55 us; skew 500 ppm
Sep 25 22:45:03 dist-test-slave-csnl krb5kdc[21485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376303, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:03.374390 21860 init.cc:377] Logged in from keytab as kudu/127.2.74.195@KRBTEST.COM (short username kudu)
I20260925 22:45:03.375574 21860 webserver.cc:460] Webserver started at http://127.2.74.195:36255/ using document root <none> and password file <none>
I20260925 22:45:03.376216 21860 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:03.376317 21860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:03.376574 21860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:03.379060 21860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-2/data/instance:
uuid: "b703738b988c453fa1fa7bedccf253bd"
format_stamp: "Formatted at 2026-09-25 22:45:03 on dist-test-slave-csnl"
server_key: "5e97e9237b7647abe4a5975c5ba70e1e"
server_key_iv: "3a3cb6e3d482b42e3e92f83081ab9aaf"
server_key_version: "encryptionkey@0"
I20260925 22:45:03.379774 21860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance:
uuid: "b703738b988c453fa1fa7bedccf253bd"
format_stamp: "Formatted at 2026-09-25 22:45:03 on dist-test-slave-csnl"
server_key: "5e97e9237b7647abe4a5975c5ba70e1e"
server_key_iv: "3a3cb6e3d482b42e3e92f83081ab9aaf"
server_key_version: "encryptionkey@0"
I20260925 22:45:03.384156 21860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:45:03.387169 21875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:03.388665 21860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:45:03.388862 21860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "b703738b988c453fa1fa7bedccf253bd"
format_stamp: "Formatted at 2026-09-25 22:45:03 on dist-test-slave-csnl"
server_key: "5e97e9237b7647abe4a5975c5ba70e1e"
server_key_iv: "3a3cb6e3d482b42e3e92f83081ab9aaf"
server_key_version: "encryptionkey@0"
I20260925 22:45:03.389012 21860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:03.403882 21860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:03.407557 21860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:03.407810 21860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:03.408490 21860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:03.409432 21860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:03.409521 21860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:03.409600 21860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:03.409643 21860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:03.419776 21860 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:42661
I20260925 22:45:03.419802 21988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:42661 every 8 connection(s)
I20260925 22:45:03.420784 21860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:45:03.423834 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 21860
I20260925 22:45:03.423996 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance
I20260925 22:45:03.424324 2347 external_mini_cluster.cc:1428] Setting key 74bdc309515c6d81ce8fbd76718d2434
Sep 25 22:45:03 dist-test-slave-csnl krb5kdc[21485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376303, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.195@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:45:03.435446 21569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:03.422923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:37069 (local address 127.2.74.254:33957)
0925 22:45:03.423100 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:03.423104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:03.424264 (+ 1160us) server_negotiation.cc:407] Connection header received
0925 22:45:03.425761 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:03.425765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:03.425823 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:03.425931 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:03.427249 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.427793 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.428992 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.429227 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.429341 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:03.431946 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:03.431955 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:03.431958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:03.431986 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:03.433288 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:03.433824 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:03.433829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:03.433831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:03.433908 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:03.434265 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:03.434268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:03.434270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:03.434459 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:03.434576 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:03.435135 (+ 559us) server_negotiation.cc:299] Negotiation successful
0925 22:45:03.435271 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 22:45:03.436313 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:03.423375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:33957 (local address 127.2.74.195:37069)
0925 22:45:03.424084 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:03.424149 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:03.425534 (+ 1385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:03.426130 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:03.426137 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:03.426520 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:03.427097 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:03.427113 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.427928 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.427931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:03.428847 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:03.428857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.429401 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.429405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:03.429594 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:03.430194 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:03.430211 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:03.431758 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:03.433412 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:03.433418 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:03.433435 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:03.433705 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:03.434009 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:03.434013 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:03.434015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:03.434162 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:03.434573 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:03.434580 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:03.434838 (+ 258us) client_negotiation.cc:769] Sending connection context
0925 22:45:03.435078 (+ 240us) client_negotiation.cc:240] Negotiation successful
0925 22:45:03.435425 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":162,"threads_started":1}
I20260925 22:45:03.437776 21989 heartbeater.cc:347] Connected to a master server at 127.2.74.254:33957
I20260925 22:45:03.438077 21989 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:03.438737 21989 heartbeater.cc:511] Master 127.2.74.254:33957 requested a full tablet report, sending...
I20260925 22:45:03.439806 21530 ts_manager.cc:194] Registered new tserver with Master: b703738b988c453fa1fa7bedccf253bd (127.2.74.195:42661)
I20260925 22:45:03.440388 21530 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.74.195@KRBTEST.COM'} at 127.2.74.195:37069
I20260925 22:45:03.450495 2347 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:45:03.460843 21569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:03.452650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35584 (local address 127.2.74.254:33957)
0925 22:45:03.452839 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:03.452844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:03.452908 (+ 64us) server_negotiation.cc:407] Connection header received
0925 22:45:03.453079 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:03.453082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:03.453138 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:03.453210 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:03.453981 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.454512 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.455520 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.455743 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.455840 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:03.457305 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:03.457309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:03.457311 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:03.457338 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:03.458847 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:03.459458 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:03.459461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:03.459463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:03.459558 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:03.459866 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:03.459868 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:03.459869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:03.460118 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:03.460228 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:03.460508 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 22:45:03.460622 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 22:45:03.464416 21530 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:45:03.467743 21530 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:45:03.483711 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:03.477918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:41769 (local address 127.0.0.1:36396)
0925 22:45:03.478923 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:03.478937 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:03.479065 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:03.479630 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:03.479633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:03.479649 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:03.479718 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:03.479727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.480750 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.480756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:03.482087 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:03.482098 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.483093 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.483097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:03.483213 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:03.483257 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 22:45:03.483392 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 22:45:03.483534 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"mutex_wait_us":5,"thread_start_us":117,"threads_started":1}
I20260925 22:45:03.483727 22004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:03.478084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36396 (local address 127.2.74.194:41769)
0925 22:45:03.478779 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:03.478791 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:03.479048 (+ 257us) server_negotiation.cc:407] Connection header received
0925 22:45:03.479167 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:03.479171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:03.479335 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:03.479471 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:03.479848 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.480602 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.482236 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.482950 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.483089 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:03.483260 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:03.483446 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 22:45:03.483565 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":148,"threads_started":1}
I20260925 22:45:03.484570 22001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:03.477164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:43181 (local address 127.0.0.1:34648)
0925 22:45:03.478584 (+ 1420us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:03.478598 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:03.478731 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:03.479548 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:03.479551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:03.479569 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:03.479676 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:03.479683 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.481160 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.481163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:03.482701 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:03.482712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.484151 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.484156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:03.484252 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:03.484275 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:45:03.484347 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 22:45:03.484431 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1357,"mutex_wait_us":205,"thread_start_us":123,"threads_started":1}
I20260925 22:45:03.485038 22005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:03.478180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34648 (local address 127.2.74.193:43181)
0925 22:45:03.478584 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:03.478592 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:03.478675 (+ 83us) server_negotiation.cc:407] Connection header received
0925 22:45:03.479009 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:03.479017 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:03.479222 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:03.479372 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:03.479806 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.481006 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.482878 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.483786 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.483920 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:03.484081 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:03.484772 (+ 691us) server_negotiation.cc:299] Negotiation successful
0925 22:45:03.484903 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":146,"threads_started":1}
I20260925 22:45:03.485427 22002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:03.477211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50086 (local address 127.2.74.195:42661)
0925 22:45:03.477559 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:03.477572 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:03.478578 (+ 1006us) server_negotiation.cc:407] Connection header received
0925 22:45:03.478931 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:03.478940 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:03.479221 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:03.479391 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:03.479698 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.481006 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.482859 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.484291 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.484434 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:03.484605 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:03.485154 (+ 549us) server_negotiation.cc:299] Negotiation successful
0925 22:45:03.485274 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":118,"threads_started":1}
I20260925 22:45:03.485447 22003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:03.477115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:42661 (local address 127.0.0.1:50086)
0925 22:45:03.478366 (+ 1251us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:03.478469 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:03.478712 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:03.479388 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:03.479394 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:03.479418 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:03.479568 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:03.479575 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.481160 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.481163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:03.482701 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:03.482712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.484856 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.484860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:03.484957 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:03.484997 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 22:45:03.485080 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 22:45:03.485315 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"mutex_wait_us":145,"thread_start_us":88,"threads_started":1}
I20260925 22:45:03.487480 21789 tablet_service.cc:1467] Processing CreateTablet for tablet 76aacfd1e1004b3592054f3bcc35b581 (DEFAULT_TABLE table=test-table [id=f86f2a5871994b20ae35d0866e1ad1e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:03.488816 21789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76aacfd1e1004b3592054f3bcc35b581. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:03.488605 21923 tablet_service.cc:1467] Processing CreateTablet for tablet 76aacfd1e1004b3592054f3bcc35b581 (DEFAULT_TABLE table=test-table [id=f86f2a5871994b20ae35d0866e1ad1e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:03.488734 21655 tablet_service.cc:1467] Processing CreateTablet for tablet 76aacfd1e1004b3592054f3bcc35b581 (DEFAULT_TABLE table=test-table [id=f86f2a5871994b20ae35d0866e1ad1e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:03.489638 21923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76aacfd1e1004b3592054f3bcc35b581. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:03.489748 21655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76aacfd1e1004b3592054f3bcc35b581. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:03.496760 22006 tablet_bootstrap.cc:492] T 76aacfd1e1004b3592054f3bcc35b581 P b703738b988c453fa1fa7bedccf253bd: Bootstrap starting.
I20260925 22:45:03.498411 22007 tablet_bootstrap.cc:492] T 76aacfd1e1004b3592054f3bcc35b581 P 1d2fd74a25b2404494d846785ef834bf: Bootstrap starting.
I20260925 22:45:03.499352 22008 tablet_bootstrap.cc:492] T 76aacfd1e1004b3592054f3bcc35b581 P 1c66c36f472e471083edd0daffc9a5b4: Bootstrap starting.
I20260925 22:45:03.499527 22006 tablet_bootstrap.cc:654] T 76aacfd1e1004b3592054f3bcc35b581 P b703738b988c453fa1fa7bedccf253bd: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:03.500846 22007 tablet_bootstrap.cc:654] T 76aacfd1e1004b3592054f3bcc35b581 P 1d2fd74a25b2404494d846785ef834bf: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:03.500916 22006 log.cc:824] T 76aacfd1e1004b3592054f3bcc35b581 P b703738b988c453fa1fa7bedccf253bd: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:03.501747 22007 log.cc:824] T 76aacfd1e1004b3592054f3bcc35b581 P 1d2fd74a25b2404494d846785ef834bf: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:03.502022 22008 tablet_bootstrap.cc:654] T 76aacfd1e1004b3592054f3bcc35b581 P 1c66c36f472e471083edd0daffc9a5b4: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:03.502871 22008 log.cc:824] T 76aacfd1e1004b3592054f3bcc35b581 P 1c66c36f472e471083edd0daffc9a5b4: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:03.503923 22007 tablet_bootstrap.cc:492] T 76aacfd1e1004b3592054f3bcc35b581 P 1d2fd74a25b2404494d846785ef834bf: No bootstrap required, opened a new log
I20260925 22:45:03.503927 22006 tablet_bootstrap.cc:492] T 76aacfd1e1004b3592054f3bcc35b581 P b703738b988c453fa1fa7bedccf253bd: No bootstrap required, opened a new log
I20260925 22:45:03.504145 22006 ts_tablet_manager.cc:1397] T 76aacfd1e1004b3592054f3bcc35b581 P b703738b988c453fa1fa7bedccf253bd: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 22:45:03.504145 22007 ts_tablet_manager.cc:1397] T 76aacfd1e1004b3592054f3bcc35b581 P 1d2fd74a25b2404494d846785ef834bf: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:45:03.504993 22008 tablet_bootstrap.cc:492] T 76aacfd1e1004b3592054f3bcc35b581 P 1c66c36f472e471083edd0daffc9a5b4: No bootstrap required, opened a new log
I20260925 22:45:03.505199 22008 ts_tablet_manager.cc:1397] T 76aacfd1e1004b3592054f3bcc35b581 P 1c66c36f472e471083edd0daffc9a5b4: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260925 22:45:03.507812 22006 raft_consensus.cc:348] T 76aacfd1e1004b3592054f3bcc35b581 P b703738b988c453fa1fa7bedccf253bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b703738b988c453fa1fa7bedccf253bd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 42661 } } peers { permanent_uuid: "1d2fd74a25b2404494d846785ef834bf" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43181 } } peers { permanent_uuid: "1c66c36f472e471083edd0daffc9a5b4" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 41769 } }
I20260925 22:45:03.508121 22006 raft_consensus.cc:374] T 76aacfd1e1004b3592054f3bcc35b581 P b703738b988c453fa1fa7bedccf253bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:03.508184 22006 raft_consensus.cc:729] T 76aacfd1e1004b3592054f3bcc35b581 P b703738b988c453fa1fa7bedccf253bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b703738b988c453fa1fa7bedccf253bd, State: Initialized, Role: FOLLOWER
I20260925 22:45:03.508653 22006 consensus_queue.cc:260] T 76aacfd1e1004b3592054f3bcc35b581 P b703738b988c453fa1fa7bedccf253bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b703738b988c453fa1fa7bedccf253bd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 42661 } } peers { permanent_uuid: "1d2fd74a25b2404494d846785ef834bf" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43181 } } peers { permanent_uuid: "1c66c36f472e471083edd0daffc9a5b4" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 41769 } }
I20260925 22:45:03.508747 22007 raft_consensus.cc:348] T 76aacfd1e1004b3592054f3bcc35b581 P 1d2fd74a25b2404494d846785ef834bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b703738b988c453fa1fa7bedccf253bd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 42661 } } peers { permanent_uuid: "1d2fd74a25b2404494d846785ef834bf" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43181 } } peers { permanent_uuid: "1c66c36f472e471083edd0daffc9a5b4" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 41769 } }
I20260925 22:45:03.509022 22007 raft_consensus.cc:374] T 76aacfd1e1004b3592054f3bcc35b581 P 1d2fd74a25b2404494d846785ef834bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:03.509086 22007 raft_consensus.cc:729] T 76aacfd1e1004b3592054f3bcc35b581 P 1d2fd74a25b2404494d846785ef834bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d2fd74a25b2404494d846785ef834bf, State: Initialized, Role: FOLLOWER
I20260925 22:45:03.509188 22008 raft_consensus.cc:348] T 76aacfd1e1004b3592054f3bcc35b581 P 1c66c36f472e471083edd0daffc9a5b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b703738b988c453fa1fa7bedccf253bd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 42661 } } peers { permanent_uuid: "1d2fd74a25b2404494d846785ef834bf" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43181 } } peers { permanent_uuid: "1c66c36f472e471083edd0daffc9a5b4" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 41769 } }
I20260925 22:45:03.509443 22008 raft_consensus.cc:374] T 76aacfd1e1004b3592054f3bcc35b581 P 1c66c36f472e471083edd0daffc9a5b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:03.509509 22008 raft_consensus.cc:729] T 76aacfd1e1004b3592054f3bcc35b581 P 1c66c36f472e471083edd0daffc9a5b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c66c36f472e471083edd0daffc9a5b4, State: Initialized, Role: FOLLOWER
I20260925 22:45:03.509516 21989 heartbeater.cc:503] Master 127.2.74.254:33957 was elected leader, sending a full tablet report...
I20260925 22:45:03.509562 22007 consensus_queue.cc:260] T 76aacfd1e1004b3592054f3bcc35b581 P 1d2fd74a25b2404494d846785ef834bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b703738b988c453fa1fa7bedccf253bd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 42661 } } peers { permanent_uuid: "1d2fd74a25b2404494d846785ef834bf" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43181 } } peers { permanent_uuid: "1c66c36f472e471083edd0daffc9a5b4" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 41769 } }
I20260925 22:45:03.509960 22008 consensus_queue.cc:260] T 76aacfd1e1004b3592054f3bcc35b581 P 1c66c36f472e471083edd0daffc9a5b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b703738b988c453fa1fa7bedccf253bd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 42661 } } peers { permanent_uuid: "1d2fd74a25b2404494d846785ef834bf" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43181 } } peers { permanent_uuid: "1c66c36f472e471083edd0daffc9a5b4" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 41769 } }
I20260925 22:45:03.510300 21721 heartbeater.cc:503] Master 127.2.74.254:33957 was elected leader, sending a full tablet report...
I20260925 22:45:03.510596 22007 ts_tablet_manager.cc:1428] T 76aacfd1e1004b3592054f3bcc35b581 P 1d2fd74a25b2404494d846785ef834bf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:45:03.510979 22006 ts_tablet_manager.cc:1428] T 76aacfd1e1004b3592054f3bcc35b581 P b703738b988c453fa1fa7bedccf253bd: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 22:45:03.511571 22008 ts_tablet_manager.cc:1428] T 76aacfd1e1004b3592054f3bcc35b581 P 1c66c36f472e471083edd0daffc9a5b4: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260925 22:45:03.511600 21855 heartbeater.cc:503] Master 127.2.74.254:33957 was elected leader, sending a full tablet report...
I20260925 22:45:03.568255 22013 raft_consensus.cc:482] T 76aacfd1e1004b3592054f3bcc35b581 P 1d2fd74a25b2404494d846785ef834bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:45:03.568507 22013 raft_consensus.cc:504] T 76aacfd1e1004b3592054f3bcc35b581 P 1d2fd74a25b2404494d846785ef834bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b703738b988c453fa1fa7bedccf253bd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 42661 } } peers { permanent_uuid: "1d2fd74a25b2404494d846785ef834bf" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43181 } } peers { permanent_uuid: "1c66c36f472e471083edd0daffc9a5b4" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 41769 } }
I20260925 22:45:03.569690 22013 leader_election.cc:288] T 76aacfd1e1004b3592054f3bcc35b581 P 1d2fd74a25b2404494d846785ef834bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b703738b988c453fa1fa7bedccf253bd (127.2.74.195:42661), 1c66c36f472e471083edd0daffc9a5b4 (127.2.74.194:41769)
I20260925 22:45:03.573998 22004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:03.570077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:40491 (local address 127.2.74.194:41769)
0925 22:45:03.570227 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:03.570232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:03.570511 (+ 279us) server_negotiation.cc:407] Connection header received
0925 22:45:03.570680 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:03.570683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:03.570734 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:03.570821 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:03.571206 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.571708 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.572860 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.573364 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.573497 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:03.573550 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:03.573772 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 22:45:03.573852 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 22:45:03.574010 22002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:03.570148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:40201 (local address 127.2.74.195:42661)
0925 22:45:03.570298 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:03.570304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:03.570627 (+ 323us) server_negotiation.cc:407] Connection header received
0925 22:45:03.570780 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:03.570785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:03.570833 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:03.570927 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:03.571206 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.571706 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.572872 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.573346 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.573472 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:03.573527 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:03.573773 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 22:45:03.573856 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 22:45:03.574034 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:03.569987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:42661 (local address 127.2.74.193:40201)
0925 22:45:03.570520 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:03.570531 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:03.570661 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:03.570937 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:03.570940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:03.570980 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:03.571100 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:03.571105 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.571815 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.571821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:03.572743 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:03.572753 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.573471 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.573475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:03.573576 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:03.573668 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 22:45:03.573756 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 22:45:03.573872 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":139,"thread_start_us":143,"threads_started":1}
I20260925 22:45:03.574527 22016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:03.569978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:41769 (local address 127.2.74.193:40491)
0925 22:45:03.570396 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:03.570412 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:03.570559 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:03.570831 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:03.570834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:03.570849 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:03.571073 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:03.571080 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.571818 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.571824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:03.572736 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:03.572745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.573490 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.573493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:03.573622 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:03.573664 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 22:45:03.573753 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 22:45:03.573869 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":39,"thread_start_us":158,"threads_started":1}
I20260925 22:45:03.574849 21943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76aacfd1e1004b3592054f3bcc35b581" candidate_uuid: "1d2fd74a25b2404494d846785ef834bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b703738b988c453fa1fa7bedccf253bd" is_pre_election: true
I20260925 22:45:03.575238 21943 raft_consensus.cc:2454] T 76aacfd1e1004b3592054f3bcc35b581 P b703738b988c453fa1fa7bedccf253bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d2fd74a25b2404494d846785ef834bf in term 0.
I20260925 22:45:03.575353 21809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76aacfd1e1004b3592054f3bcc35b581" candidate_uuid: "1d2fd74a25b2404494d846785ef834bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c66c36f472e471083edd0daffc9a5b4" is_pre_election: true
I20260925 22:45:03.575688 21809 raft_consensus.cc:2454] T 76aacfd1e1004b3592054f3bcc35b581 P 1c66c36f472e471083edd0daffc9a5b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d2fd74a25b2404494d846785ef834bf in term 0.
I20260925 22:45:03.575755 21608 leader_election.cc:302] T 76aacfd1e1004b3592054f3bcc35b581 P 1d2fd74a25b2404494d846785ef834bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d2fd74a25b2404494d846785ef834bf, b703738b988c453fa1fa7bedccf253bd; no voters:
I20260925 22:45:03.576093 22013 raft_consensus.cc:2781] T 76aacfd1e1004b3592054f3bcc35b581 P 1d2fd74a25b2404494d846785ef834bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:45:03.576196 22013 raft_consensus.cc:482] T 76aacfd1e1004b3592054f3bcc35b581 P 1d2fd74a25b2404494d846785ef834bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:45:03.576241 22013 raft_consensus.cc:3037] T 76aacfd1e1004b3592054f3bcc35b581 P 1d2fd74a25b2404494d846785ef834bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:03.577299 22013 raft_consensus.cc:504] T 76aacfd1e1004b3592054f3bcc35b581 P 1d2fd74a25b2404494d846785ef834bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b703738b988c453fa1fa7bedccf253bd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 42661 } } peers { permanent_uuid: "1d2fd74a25b2404494d846785ef834bf" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43181 } } peers { permanent_uuid: "1c66c36f472e471083edd0daffc9a5b4" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 41769 } }
I20260925 22:45:03.577667 22013 leader_election.cc:288] T 76aacfd1e1004b3592054f3bcc35b581 P 1d2fd74a25b2404494d846785ef834bf [CANDIDATE]: Term 1 election: Requested vote from peers b703738b988c453fa1fa7bedccf253bd (127.2.74.195:42661), 1c66c36f472e471083edd0daffc9a5b4 (127.2.74.194:41769)
I20260925 22:45:03.578013 21943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76aacfd1e1004b3592054f3bcc35b581" candidate_uuid: "1d2fd74a25b2404494d846785ef834bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b703738b988c453fa1fa7bedccf253bd"
I20260925 22:45:03.578068 21809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76aacfd1e1004b3592054f3bcc35b581" candidate_uuid: "1d2fd74a25b2404494d846785ef834bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c66c36f472e471083edd0daffc9a5b4"
I20260925 22:45:03.578169 21943 raft_consensus.cc:3037] T 76aacfd1e1004b3592054f3bcc35b581 P b703738b988c453fa1fa7bedccf253bd [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:03.578210 21809 raft_consensus.cc:3037] T 76aacfd1e1004b3592054f3bcc35b581 P 1c66c36f472e471083edd0daffc9a5b4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:03.579365 21943 raft_consensus.cc:2454] T 76aacfd1e1004b3592054f3bcc35b581 P b703738b988c453fa1fa7bedccf253bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d2fd74a25b2404494d846785ef834bf in term 1.
I20260925 22:45:03.579371 21809 raft_consensus.cc:2454] T 76aacfd1e1004b3592054f3bcc35b581 P 1c66c36f472e471083edd0daffc9a5b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d2fd74a25b2404494d846785ef834bf in term 1.
I20260925 22:45:03.579726 21608 leader_election.cc:302] T 76aacfd1e1004b3592054f3bcc35b581 P 1d2fd74a25b2404494d846785ef834bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1c66c36f472e471083edd0daffc9a5b4, 1d2fd74a25b2404494d846785ef834bf, b703738b988c453fa1fa7bedccf253bd; no voters:
I20260925 22:45:03.579950 22013 raft_consensus.cc:2781] T 76aacfd1e1004b3592054f3bcc35b581 P 1d2fd74a25b2404494d846785ef834bf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:03.580217 22013 raft_consensus.cc:686] T 76aacfd1e1004b3592054f3bcc35b581 P 1d2fd74a25b2404494d846785ef834bf [term 1 LEADER]: Becoming Leader. State: Replica: 1d2fd74a25b2404494d846785ef834bf, State: Running, Role: LEADER
I20260925 22:45:03.580574 22013 consensus_queue.cc:237] T 76aacfd1e1004b3592054f3bcc35b581 P 1d2fd74a25b2404494d846785ef834bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b703738b988c453fa1fa7bedccf253bd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 42661 } } peers { permanent_uuid: "1d2fd74a25b2404494d846785ef834bf" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43181 } } peers { permanent_uuid: "1c66c36f472e471083edd0daffc9a5b4" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 41769 } }
I20260925 22:45:03.584275 21528 catalog_manager.cc:5495] T 76aacfd1e1004b3592054f3bcc35b581 P 1d2fd74a25b2404494d846785ef834bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d2fd74a25b2404494d846785ef834bf (127.2.74.193). New cstate: current_term: 1 leader_uuid: "1d2fd74a25b2404494d846785ef834bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b703738b988c453fa1fa7bedccf253bd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 42661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d2fd74a25b2404494d846785ef834bf" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 43181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c66c36f472e471083edd0daffc9a5b4" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 41769 } health_report { overall_health: UNKNOWN } } }
I20260925 22:45:03.615371 22005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:03.611610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34660 (local address 127.2.74.193:43181)
0925 22:45:03.611740 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:03.611743 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:03.611757 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:45:03.611842 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:03.611844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:03.611927 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:03.612026 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:45:03.612275 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.612802 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.613806 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.614043 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.614127 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:03.614455 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:45:03.614989 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:45:03.615083 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:03.615189 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 22:45:03.615248 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:45:03.621621 21943 raft_consensus.cc:1261] T 76aacfd1e1004b3592054f3bcc35b581 P b703738b988c453fa1fa7bedccf253bd [term 1 FOLLOWER]: Refusing update from remote peer 1d2fd74a25b2404494d846785ef834bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:03.621621 21809 raft_consensus.cc:1261] T 76aacfd1e1004b3592054f3bcc35b581 P 1c66c36f472e471083edd0daffc9a5b4 [term 1 FOLLOWER]: Refusing update from remote peer 1d2fd74a25b2404494d846785ef834bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:03.622447 22017 consensus_queue.cc:1035] T 76aacfd1e1004b3592054f3bcc35b581 P 1d2fd74a25b2404494d846785ef834bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c66c36f472e471083edd0daffc9a5b4" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 41769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:03.622666 22013 consensus_queue.cc:1035] T 76aacfd1e1004b3592054f3bcc35b581 P 1d2fd74a25b2404494d846785ef834bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b703738b988c453fa1fa7bedccf253bd" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 42661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:03.629868 22020 mvcc.cc:205] Tried to move back new op lower bound from 7333381339626467328 to 7333381339469213696. Current Snapshot: MvccSnapshot[applied={T|T < 7333381339626467328}]
I20260925 22:45:03.632112 22022 mvcc.cc:205] Tried to move back new op lower bound from 7333381339626467328 to 7333381339469213696. Current Snapshot: MvccSnapshot[applied={T|T < 7333381339626467328}]
I20260925 22:45:03.632481 22021 mvcc.cc:205] Tried to move back new op lower bound from 7333381339626467328 to 7333381339469213696. Current Snapshot: MvccSnapshot[applied={T|T < 7333381339626467328}]
I20260925 22:45:03.639385 21528 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:45:03.639616 21528 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:45:03.643407 21528 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f66bb5c9e5b40aa9c97c2f0c3291146: Sending DeleteTablet for 3 replicas of tablet 76aacfd1e1004b3592054f3bcc35b581
I20260925 22:45:03.644377 21655 tablet_service.cc:1514] Processing DeleteTablet for tablet 76aacfd1e1004b3592054f3bcc35b581 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:03 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:34648
I20260925 22:45:03.644388 21923 tablet_service.cc:1514] Processing DeleteTablet for tablet 76aacfd1e1004b3592054f3bcc35b581 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:03 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:50086
I20260925 22:45:03.644521 21789 tablet_service.cc:1514] Processing DeleteTablet for tablet 76aacfd1e1004b3592054f3bcc35b581 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:03 UTC) from {username='kudu', principal='kudu/127.2.74.254@KRBTEST.COM'} at 127.0.0.1:36396
I20260925 22:45:03.644991 22030 tablet_replica.cc:330] T 76aacfd1e1004b3592054f3bcc35b581 P b703738b988c453fa1fa7bedccf253bd: stopping tablet replica
I20260925 22:45:03.645196 22031 tablet_replica.cc:330] T 76aacfd1e1004b3592054f3bcc35b581 P 1d2fd74a25b2404494d846785ef834bf: stopping tablet replica
I20260925 22:45:03.645368 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 21592
I20260925 22:45:03.645366 22030 raft_consensus.cc:2229] T 76aacfd1e1004b3592054f3bcc35b581 P b703738b988c453fa1fa7bedccf253bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:45:03.645524 22031 raft_consensus.cc:2229] T 76aacfd1e1004b3592054f3bcc35b581 P 1d2fd74a25b2404494d846785ef834bf [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:45:03.645738 22030 raft_consensus.cc:2258] T 76aacfd1e1004b3592054f3bcc35b581 P b703738b988c453fa1fa7bedccf253bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:45:03.647563 22030 ts_tablet_manager.cc:1905] T 76aacfd1e1004b3592054f3bcc35b581 P b703738b988c453fa1fa7bedccf253bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:45:03.650427 22030 ts_tablet_manager.cc:1918] T 76aacfd1e1004b3592054f3bcc35b581 P b703738b988c453fa1fa7bedccf253bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:45:03.650530 22030 log.cc:1196] T 76aacfd1e1004b3592054f3bcc35b581 P b703738b988c453fa1fa7bedccf253bd: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-2/wal/wals/76aacfd1e1004b3592054f3bcc35b581
I20260925 22:45:03.650763 22032 tablet_replica.cc:330] T 76aacfd1e1004b3592054f3bcc35b581 P 1c66c36f472e471083edd0daffc9a5b4: stopping tablet replica
I20260925 22:45:03.650854 22030 ts_tablet_manager.cc:1939] T 76aacfd1e1004b3592054f3bcc35b581 P b703738b988c453fa1fa7bedccf253bd: Deleting consensus metadata
I20260925 22:45:03.651152 22032 raft_consensus.cc:2229] T 76aacfd1e1004b3592054f3bcc35b581 P 1c66c36f472e471083edd0daffc9a5b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:45:03.651511 22032 raft_consensus.cc:2258] T 76aacfd1e1004b3592054f3bcc35b581 P 1c66c36f472e471083edd0daffc9a5b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:45:03.652576 21514 catalog_manager.cc:4841] TS b703738b988c453fa1fa7bedccf253bd (127.2.74.195:42661): tablet 76aacfd1e1004b3592054f3bcc35b581 (table test-table [id=f86f2a5871994b20ae35d0866e1ad1e5]) successfully deleted
I20260925 22:45:03.653007 22032 ts_tablet_manager.cc:1905] T 76aacfd1e1004b3592054f3bcc35b581 P 1c66c36f472e471083edd0daffc9a5b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:45:03.656625 22032 ts_tablet_manager.cc:1918] T 76aacfd1e1004b3592054f3bcc35b581 P 1c66c36f472e471083edd0daffc9a5b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:45:03.656751 22032 log.cc:1196] T 76aacfd1e1004b3592054f3bcc35b581 P 1c66c36f472e471083edd0daffc9a5b4: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376047580784-2347-0/minicluster-data/ts-1/wal/wals/76aacfd1e1004b3592054f3bcc35b581
I20260925 22:45:03.657106 22032 ts_tablet_manager.cc:1939] T 76aacfd1e1004b3592054f3bcc35b581 P 1c66c36f472e471083edd0daffc9a5b4: Deleting consensus metadata
W20260925 22:45:03.657148 21515 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.74.193:43181 (error 108)
I20260925 22:45:03.658082 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 21726
I20260925 22:45:03.658598 22003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:03.658164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:43181 (local address 127.0.0.1:34666)
0925 22:45:03.658403 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:03.658493 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.193:43181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260925 22:45:03.658851 21515 catalog_manager.cc:4584] TS 1d2fd74a25b2404494d846785ef834bf (127.2.74.193:43181): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 76aacfd1e1004b3592054f3bcc35b581: Network error: Client connection negotiation failed: client connection to 127.2.74.193:43181: connect: Connection refused (error 111)
I20260925 22:45:03.664499 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 21860
I20260925 22:45:03.666729 22003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:03.664869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:41769 (local address 127.0.0.1:36404)
0925 22:45:03.666574 (+ 1705us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:03.666645 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.194:41769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1576}
I20260925 22:45:03.670840 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 21498
2026-09-25T22:45:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2403 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T22:45:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:03Z Disabled control of system clock
I20260925 22:45:03.702482 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:45427
--webserver_interface=127.2.74.254
--webserver_port=0
--builtin_ntp_servers=127.2.74.212:43581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:45427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:45:03.818809 22037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:03.819149 22037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:03.819190 22037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:03.823415 22037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:03.823488 22037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:03.823504 22037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:03.823522 22037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:03.823537 22037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:03.828856 22037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:43581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:45427
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:45427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:03.829979 22037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:03.831427 22037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:03.840603 22045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:03.840646 22037 server_base.cc:1034] running on GCE node
W20260925 22:45:03.840607 22043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:03.840603 22042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:03.841450 22037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:03.842685 22037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:03.843868 22037 hybrid_clock.cc:648] HybridClock initialized: now 1790376303843847 us; error 41 us; skew 500 ppm
I20260925 22:45:03.845999 22037 webserver.cc:460] Webserver started at http://127.2.74.254:33695/ using document root <none> and password file <none>
I20260925 22:45:03.846637 22037 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:03.846707 22037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:03.847010 22037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:03.849170 22037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "e750bf1af60e484cac51e9c46882ba67"
format_stamp: "Formatted at 2026-09-25 22:45:03 on dist-test-slave-csnl"
server_key: "451c8c85b524250579c1368796ba358e"
server_key_iv: "720d17ef648315fda2c235858f2ed8d9"
server_key_version: "encryptionkey@0"
I20260925 22:45:03.849781 22037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "e750bf1af60e484cac51e9c46882ba67"
format_stamp: "Formatted at 2026-09-25 22:45:03 on dist-test-slave-csnl"
server_key: "451c8c85b524250579c1368796ba358e"
server_key_iv: "720d17ef648315fda2c235858f2ed8d9"
server_key_version: "encryptionkey@0"
I20260925 22:45:03.853761 22037 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 22:45:03.856563 22051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:03.857976 22037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:45:03.858153 22037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "e750bf1af60e484cac51e9c46882ba67"
format_stamp: "Formatted at 2026-09-25 22:45:03 on dist-test-slave-csnl"
server_key: "451c8c85b524250579c1368796ba358e"
server_key_iv: "720d17ef648315fda2c235858f2ed8d9"
server_key_version: "encryptionkey@0"
I20260925 22:45:03.858314 22037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:03.885040 22037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:03.885856 22037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:03.886081 22037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:03.893893 22037 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:45427
I20260925 22:45:03.893853 22103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:45427 every 8 connection(s)
I20260925 22:45:03.894977 22037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:03.898329 22104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:03.899647 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 22037
I20260925 22:45:03.899807 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:45:03.900120 2347 external_mini_cluster.cc:1428] Setting key 6f36a6af9f0e0f2f53eb1cadbc901fa4
I20260925 22:45:03.904798 22104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e750bf1af60e484cac51e9c46882ba67: Bootstrap starting.
I20260925 22:45:03.907831 22104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e750bf1af60e484cac51e9c46882ba67: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:03.908715 22104 log.cc:824] T 00000000000000000000000000000000 P e750bf1af60e484cac51e9c46882ba67: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:03.910202 22107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:03.901598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47668 (local address 127.2.74.254:45427)
0925 22:45:03.902232 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:03.902247 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:03.902282 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:45:03.902932 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:03.903006 (+ 74us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:03.903431 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:03.903714 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:03.904112 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.905133 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.906400 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.906915 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.907158 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:03.908239 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:03.908245 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:03.908260 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:03.908721 (+ 461us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:03.908783 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:03.908850 (+ 67us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:03.908990 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:03.909160 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 22:45:03.909446 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":149,"threads_started":1}
I20260925 22:45:03.911010 22104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e750bf1af60e484cac51e9c46882ba67: No bootstrap required, opened a new log
I20260925 22:45:03.914852 22104 raft_consensus.cc:348] T 00000000000000000000000000000000 P e750bf1af60e484cac51e9c46882ba67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e750bf1af60e484cac51e9c46882ba67" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 45427 } }
I20260925 22:45:03.915225 22104 raft_consensus.cc:374] T 00000000000000000000000000000000 P e750bf1af60e484cac51e9c46882ba67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:03.915298 22104 raft_consensus.cc:729] T 00000000000000000000000000000000 P e750bf1af60e484cac51e9c46882ba67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e750bf1af60e484cac51e9c46882ba67, State: Initialized, Role: FOLLOWER
I20260925 22:45:03.915768 22104 consensus_queue.cc:260] T 00000000000000000000000000000000 P e750bf1af60e484cac51e9c46882ba67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e750bf1af60e484cac51e9c46882ba67" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 45427 } }
I20260925 22:45:03.915900 22104 raft_consensus.cc:388] T 00000000000000000000000000000000 P e750bf1af60e484cac51e9c46882ba67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:03.915979 22104 raft_consensus.cc:482] T 00000000000000000000000000000000 P e750bf1af60e484cac51e9c46882ba67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:03.916091 22104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e750bf1af60e484cac51e9c46882ba67 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:03.917295 22104 raft_consensus.cc:504] T 00000000000000000000000000000000 P e750bf1af60e484cac51e9c46882ba67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e750bf1af60e484cac51e9c46882ba67" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 45427 } }
I20260925 22:45:03.917627 22104 leader_election.cc:302] T 00000000000000000000000000000000 P e750bf1af60e484cac51e9c46882ba67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e750bf1af60e484cac51e9c46882ba67; no voters:
I20260925 22:45:03.917912 22104 leader_election.cc:288] T 00000000000000000000000000000000 P e750bf1af60e484cac51e9c46882ba67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:03.918062 22109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e750bf1af60e484cac51e9c46882ba67 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:03.918354 22109 raft_consensus.cc:686] T 00000000000000000000000000000000 P e750bf1af60e484cac51e9c46882ba67 [term 1 LEADER]: Becoming Leader. State: Replica: e750bf1af60e484cac51e9c46882ba67, State: Running, Role: LEADER
I20260925 22:45:03.918690 22109 consensus_queue.cc:237] T 00000000000000000000000000000000 P e750bf1af60e484cac51e9c46882ba67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e750bf1af60e484cac51e9c46882ba67" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 45427 } }
I20260925 22:45:03.919322 22104 sys_catalog.cc:564] T 00000000000000000000000000000000 P e750bf1af60e484cac51e9c46882ba67 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:03.920475 22111 sys_catalog.cc:455] T 00000000000000000000000000000000 P e750bf1af60e484cac51e9c46882ba67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e750bf1af60e484cac51e9c46882ba67. Latest consensus state: current_term: 1 leader_uuid: "e750bf1af60e484cac51e9c46882ba67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e750bf1af60e484cac51e9c46882ba67" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 45427 } } }
I20260925 22:45:03.920614 22111 sys_catalog.cc:458] T 00000000000000000000000000000000 P e750bf1af60e484cac51e9c46882ba67 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:03.920899 22110 sys_catalog.cc:455] T 00000000000000000000000000000000 P e750bf1af60e484cac51e9c46882ba67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e750bf1af60e484cac51e9c46882ba67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e750bf1af60e484cac51e9c46882ba67" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 45427 } } }
I20260925 22:45:03.921000 22110 sys_catalog.cc:458] T 00000000000000000000000000000000 P e750bf1af60e484cac51e9c46882ba67 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:03.921099 22118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:03.924796 22118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:03.930577 22118 catalog_manager.cc:1347] Generated new cluster ID: 3c44d6e21ab94b2f8e000f7b1d75d81b
I20260925 22:45:03.930675 22118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:03.946164 22118 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:03.947630 22118 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:03.956151 22118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e750bf1af60e484cac51e9c46882ba67: Generated new TSK 0
I20260925 22:45:03.957121 22118 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:45:03.960583 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.193:0
--local_ip_for_outbound_sockets=127.2.74.193
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:45427
--builtin_ntp_servers=127.2.74.212:43581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:45:04.084319 22128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:04.084709 22128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:04.084787 22128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:04.089304 22128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:04.089430 22128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:04.089538 22128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.193
I20260925 22:45:04.095098 22128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:43581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.74.193
--webserver_port=0
--tserver_master_addrs=127.2.74.254:45427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.193
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:04.096321 22128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:04.097838 22128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:04.108197 22136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:04.108199 22133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:04.108253 22128 server_base.cc:1034] running on GCE node
W20260925 22:45:04.108199 22134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:04.108927 22128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:04.109615 22128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:04.110796 22128 hybrid_clock.cc:648] HybridClock initialized: now 1790376304110775 us; error 36 us; skew 500 ppm
I20260925 22:45:04.112926 22128 webserver.cc:460] Webserver started at http://127.2.74.193:39925/ using document root <none> and password file <none>
I20260925 22:45:04.113590 22128 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:04.113665 22128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:04.113934 22128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:04.116267 22128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-0/data/instance:
uuid: "32dd31b07bb64da5b1cb07bfa9de518b"
format_stamp: "Formatted at 2026-09-25 22:45:04 on dist-test-slave-csnl"
server_key: "af0fefa3ff7936dfd84e64d1b8cf8897"
server_key_iv: "5720b21393e5cb7139f23300b0d0352c"
server_key_version: "encryptionkey@0"
I20260925 22:45:04.116921 22128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance:
uuid: "32dd31b07bb64da5b1cb07bfa9de518b"
format_stamp: "Formatted at 2026-09-25 22:45:04 on dist-test-slave-csnl"
server_key: "af0fefa3ff7936dfd84e64d1b8cf8897"
server_key_iv: "5720b21393e5cb7139f23300b0d0352c"
server_key_version: "encryptionkey@0"
I20260925 22:45:04.121322 22128 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:45:04.124282 22142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:04.126013 22128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:04.126191 22128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-0/wal
uuid: "32dd31b07bb64da5b1cb07bfa9de518b"
format_stamp: "Formatted at 2026-09-25 22:45:04 on dist-test-slave-csnl"
server_key: "af0fefa3ff7936dfd84e64d1b8cf8897"
server_key_iv: "5720b21393e5cb7139f23300b0d0352c"
server_key_version: "encryptionkey@0"
I20260925 22:45:04.126338 22128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:04.164147 22128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:04.164918 22128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:04.165138 22128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:04.165767 22128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:04.166862 22128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:04.166949 22128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:04.167071 22128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:04.167125 22128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:04.176887 22128 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.193:37551
I20260925 22:45:04.176898 22255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.193:37551 every 8 connection(s)
I20260925 22:45:04.177896 22128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-0/data/info.pb
I20260925 22:45:04.178606 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 22128
I20260925 22:45:04.178767 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-0/wal/instance
I20260925 22:45:04.179210 2347 external_mini_cluster.cc:1428] Setting key 8525c589d5531cf5f2644efb92e5a2bd
I20260925 22:45:04.181499 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.194:0
--local_ip_for_outbound_sockets=127.2.74.194
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:45427
--builtin_ntp_servers=127.2.74.212:43581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:45:04.191120 22107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:04.180071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.193:44755 (local address 127.2.74.254:45427)
0925 22:45:04.180217 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:04.180222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:04.181679 (+ 1457us) server_negotiation.cc:407] Connection header received
0925 22:45:04.183821 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:04.183825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:04.183913 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:04.184000 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:04.185420 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.186116 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.188321 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.188598 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.188700 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.190357 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:04.190360 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:04.190363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:04.190378 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:04.190403 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:04.190411 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:04.190575 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:04.190801 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 22:45:04.190895 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:45:04.192514 22258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:04.180629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:45427 (local address 127.2.74.193:44755)
0925 22:45:04.181494 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:04.181562 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:04.183541 (+ 1979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:04.184224 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:04.184237 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:04.184372 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:04.185164 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.185189 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.186325 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.186331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.188138 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.188157 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.188810 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.188817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.189093 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.189925 (+ 832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:45:04.189930 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:04.190005 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:45:04.190124 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:04.190574 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:04.190586 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:04.190594 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 22:45:04.190761 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 22:45:04.191186 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":128,"threads_started":1}
I20260925 22:45:04.194113 22256 heartbeater.cc:347] Connected to a master server at 127.2.74.254:45427
I20260925 22:45:04.194430 22256 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:04.195112 22256 heartbeater.cc:511] Master 127.2.74.254:45427 requested a full tablet report, sending...
I20260925 22:45:04.196916 22068 ts_manager.cc:194] Registered new tserver with Master: 32dd31b07bb64da5b1cb07bfa9de518b (127.2.74.193:37551)
I20260925 22:45:04.198190 22068 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.74.193:44755
W20260925 22:45:04.303100 22259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:04.303464 22259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:04.303540 22259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:04.308101 22259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:04.308212 22259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:04.308317 22259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.194
I20260925 22:45:04.313500 22259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:43581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.74.194
--webserver_port=0
--tserver_master_addrs=127.2.74.254:45427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.194
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:04.314692 22259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:04.316219 22259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:04.326820 22265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:04.326928 22267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:04.326814 22264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:04.328388 22259 server_base.cc:1034] running on GCE node
I20260925 22:45:04.328903 22259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:04.329620 22259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:04.330984 22259 hybrid_clock.cc:648] HybridClock initialized: now 1790376304330935 us; error 35 us; skew 500 ppm
I20260925 22:45:04.333002 22259 webserver.cc:460] Webserver started at http://127.2.74.194:33963/ using document root <none> and password file <none>
I20260925 22:45:04.333600 22259 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:04.333673 22259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:04.333855 22259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:04.336210 22259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-1/data/instance:
uuid: "d567fe99f22348adb0c0d9833cbdd1ea"
format_stamp: "Formatted at 2026-09-25 22:45:04 on dist-test-slave-csnl"
server_key: "b083b6c229964561e391b9a6e818100a"
server_key_iv: "bf954e9837d8a107a277c455167d7abb"
server_key_version: "encryptionkey@0"
I20260925 22:45:04.336776 22259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance:
uuid: "d567fe99f22348adb0c0d9833cbdd1ea"
format_stamp: "Formatted at 2026-09-25 22:45:04 on dist-test-slave-csnl"
server_key: "b083b6c229964561e391b9a6e818100a"
server_key_iv: "bf954e9837d8a107a277c455167d7abb"
server_key_version: "encryptionkey@0"
I20260925 22:45:04.340988 22259 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:45:04.343699 22273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:04.345084 22259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:04.345218 22259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-1/wal
uuid: "d567fe99f22348adb0c0d9833cbdd1ea"
format_stamp: "Formatted at 2026-09-25 22:45:04 on dist-test-slave-csnl"
server_key: "b083b6c229964561e391b9a6e818100a"
server_key_iv: "bf954e9837d8a107a277c455167d7abb"
server_key_version: "encryptionkey@0"
I20260925 22:45:04.345330 22259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:04.363296 22259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:04.364029 22259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:04.364199 22259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:04.364794 22259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:04.365775 22259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:04.365828 22259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:04.365870 22259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:04.365886 22259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:04.375308 22259 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.194:37029
I20260925 22:45:04.375352 22386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.194:37029 every 8 connection(s)
I20260925 22:45:04.376294 22259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-1/data/info.pb
I20260925 22:45:04.379747 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 22259
I20260925 22:45:04.379904 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-1/wal/instance
I20260925 22:45:04.380201 2347 external_mini_cluster.cc:1428] Setting key 9aa99ce803bc6f4bc9bb938cc2323a20
I20260925 22:45:04.382266 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.74.195:0
--local_ip_for_outbound_sockets=127.2.74.195
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:45427
--builtin_ntp_servers=127.2.74.212:43581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:45:04.387259 22107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:04.378337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.194:44319 (local address 127.2.74.254:45427)
0925 22:45:04.378515 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:04.378518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:04.379689 (+ 1171us) server_negotiation.cc:407] Connection header received
0925 22:45:04.381405 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:04.381409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:04.381466 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:04.381555 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:04.382627 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.383241 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.385041 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.385274 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.385413 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.386667 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:04.386671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:04.386673 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:04.386696 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:04.386717 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:04.386725 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:04.386804 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:04.387020 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 22:45:04.387101 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 22:45:04.388200 22389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:04.378768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:45427 (local address 127.2.74.194:44319)
0925 22:45:04.379518 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:04.379562 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:04.381174 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:04.381757 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:04.381764 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:04.381865 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:04.382454 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.382481 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.383376 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.383383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.384904 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.384917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.385478 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.385483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.385669 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.386359 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:45:04.386361 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:04.386410 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:45:04.386487 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:04.386818 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:04.386828 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:04.386834 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:45:04.387008 (+ 174us) client_negotiation.cc:240] Negotiation successful
0925 22:45:04.387330 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":185,"threads_started":1}
I20260925 22:45:04.389704 22387 heartbeater.cc:347] Connected to a master server at 127.2.74.254:45427
I20260925 22:45:04.389981 22387 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:04.390581 22387 heartbeater.cc:511] Master 127.2.74.254:45427 requested a full tablet report, sending...
I20260925 22:45:04.391919 22068 ts_manager.cc:194] Registered new tserver with Master: d567fe99f22348adb0c0d9833cbdd1ea (127.2.74.194:37029)
I20260925 22:45:04.392549 22068 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.74.194:44319
W20260925 22:45:04.501955 22390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:04.502731 22390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:04.502936 22390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:04.507805 22390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:04.507906 22390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:04.507999 22390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.74.195
I20260925 22:45:04.513420 22390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:43581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.74.195
--webserver_port=0
--tserver_master_addrs=127.2.74.254:45427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.74.195
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:04.514645 22390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:04.516191 22390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:04.526753 22395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:04.526753 22396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:04.526769 22398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:04.527051 22390 server_base.cc:1034] running on GCE node
I20260925 22:45:04.527539 22390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:04.528203 22390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:04.529374 22390 hybrid_clock.cc:648] HybridClock initialized: now 1790376304529343 us; error 48 us; skew 500 ppm
I20260925 22:45:04.531479 22390 webserver.cc:460] Webserver started at http://127.2.74.195:43545/ using document root <none> and password file <none>
I20260925 22:45:04.532092 22390 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:04.532157 22390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:04.532383 22390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:04.534574 22390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-2/data/instance:
uuid: "c2b0fde054e049c999465cef407da18a"
format_stamp: "Formatted at 2026-09-25 22:45:04 on dist-test-slave-csnl"
server_key: "830fef92fc30b2ad0b62a30ae918115a"
server_key_iv: "4d0573be57f50246eb038d97cb365ef3"
server_key_version: "encryptionkey@0"
I20260925 22:45:04.535209 22390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance:
uuid: "c2b0fde054e049c999465cef407da18a"
format_stamp: "Formatted at 2026-09-25 22:45:04 on dist-test-slave-csnl"
server_key: "830fef92fc30b2ad0b62a30ae918115a"
server_key_iv: "4d0573be57f50246eb038d97cb365ef3"
server_key_version: "encryptionkey@0"
I20260925 22:45:04.539191 22390 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:45:04.541951 22404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:04.543462 22390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:04.543625 22390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-2/wal
uuid: "c2b0fde054e049c999465cef407da18a"
format_stamp: "Formatted at 2026-09-25 22:45:04 on dist-test-slave-csnl"
server_key: "830fef92fc30b2ad0b62a30ae918115a"
server_key_iv: "4d0573be57f50246eb038d97cb365ef3"
server_key_version: "encryptionkey@0"
I20260925 22:45:04.543762 22390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:04.557217 22390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:04.557983 22390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:04.558192 22390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:04.558831 22390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:04.559883 22390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:04.559966 22390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:04.560038 22390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:04.560091 22390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:04.569366 22390 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.195:33609
I20260925 22:45:04.569407 22517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.195:33609 every 8 connection(s)
I20260925 22:45:04.570367 22390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-2/data/info.pb
I20260925 22:45:04.579617 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 22390
I20260925 22:45:04.579787 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-2/wal/instance
I20260925 22:45:04.580133 2347 external_mini_cluster.cc:1428] Setting key a925c5b8d61a988721488920c3323b70
I20260925 22:45:04.580888 22107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:04.572389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:39009 (local address 127.2.74.254:45427)
0925 22:45:04.572582 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:04.572587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:04.573850 (+ 1263us) server_negotiation.cc:407] Connection header received
0925 22:45:04.575433 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:04.575436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:04.575494 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:04.575584 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:04.576685 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.577241 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.578535 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.578779 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.578878 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.580344 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:04.580348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:04.580350 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:04.580366 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:04.580387 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:04.580396 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:04.580479 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:04.580662 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 22:45:04.580740 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 22:45:04.582201 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:04.572831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:45427 (local address 127.2.74.195:39009)
0925 22:45:04.573658 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:04.573707 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:04.575158 (+ 1451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:04.575822 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:04.575830 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:04.575932 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:04.576515 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.576531 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.577378 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.577381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.578360 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.578371 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.579024 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.579030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.579211 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.579992 (+ 781us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:45:04.579994 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:04.580042 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:45:04.580122 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:04.580495 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:04.580507 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:04.580518 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 22:45:04.580653 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 22:45:04.581006 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":133,"threads_started":1}
I20260925 22:45:04.583608 22518 heartbeater.cc:347] Connected to a master server at 127.2.74.254:45427
I20260925 22:45:04.583890 22518 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:04.584501 22518 heartbeater.cc:511] Master 127.2.74.254:45427 requested a full tablet report, sending...
I20260925 22:45:04.585721 22068 ts_manager.cc:194] Registered new tserver with Master: c2b0fde054e049c999465cef407da18a (127.2.74.195:33609)
I20260925 22:45:04.586370 22068 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.74.195:39009
I20260925 22:45:04.595094 2347 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:45:04.602008 22107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:04.597633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47678 (local address 127.2.74.254:45427)
0925 22:45:04.597837 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:04.597842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:04.597987 (+ 145us) server_negotiation.cc:407] Connection header received
0925 22:45:04.598175 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:04.598179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:04.598236 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:04.598355 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:04.598678 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.599307 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.600214 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.600469 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.600593 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.601502 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:04.601506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:04.601508 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:04.601522 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:04.601544 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:04.601552 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:04.601641 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:04.601763 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 22:45:04.601832 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 22:45:04.605240 22068 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:45:04.609040 22068 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:45:04.625880 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:04.619963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33936 (local address 127.2.74.195:33609)
0925 22:45:04.620258 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:04.620269 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:04.620371 (+ 102us) server_negotiation.cc:407] Connection header received
0925 22:45:04.620497 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:04.620503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:04.620739 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:04.621426 (+ 687us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:04.621580 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.622730 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.624477 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.625089 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.625232 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.625458 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:04.625572 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 22:45:04.625683 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":87,"threads_started":1}
I20260925 22:45:04.625861 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:04.619806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.195:33609 (local address 127.0.0.1:33936)
0925 22:45:04.620223 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:04.620238 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:04.620373 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:04.620955 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:04.620961 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:04.621000 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:04.621263 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.621271 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.622900 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.622904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.624325 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.624336 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.625237 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.625241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.625357 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.625409 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 22:45:04.625540 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 22:45:04.625684 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":43,"spinlock_wait_cycles":7424,"thread_start_us":121,"threads_started":1}
I20260925 22:45:04.626276 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:04.619034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:37029 (local address 127.0.0.1:47076)
0925 22:45:04.619638 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:04.619692 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:04.619839 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:04.620956 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:04.620961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:04.621004 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:04.621269 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.621278 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.622858 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.622861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.624414 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.624424 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.625726 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.625730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.625904 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.625956 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 22:45:04.626033 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 22:45:04.626121 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":397,"spinlock_wait_cycles":5632,"thread_start_us":120,"threads_started":1}
I20260925 22:45:04.626718 22534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:04.619037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:37551 (local address 127.0.0.1:39704)
0925 22:45:04.620569 (+ 1532us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:04.620581 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:04.620673 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:04.621458 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:04.621461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:04.621484 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:04.621570 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.621582 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.622633 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.622637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.624341 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.624354 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.625726 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.625730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.625909 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.626456 (+ 547us) client_negotiation.cc:769] Sending connection context
0925 22:45:04.626526 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 22:45:04.626594 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1477,"mutex_wait_us":1608,"thread_start_us":124,"threads_started":1}
I20260925 22:45:04.627009 22531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:04.619286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39704 (local address 127.2.74.193:37551)
0925 22:45:04.619655 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:04.619670 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:04.620780 (+ 1110us) server_negotiation.cc:407] Connection header received
0925 22:45:04.620886 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:04.620893 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:04.621103 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:04.621286 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:04.621696 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.622486 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.624510 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.625355 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.625468 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.625655 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:04.626668 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0925 22:45:04.626803 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":165,"threads_started":1}
I20260925 22:45:04.627452 22530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:04.619106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47076 (local address 127.2.74.194:37029)
0925 22:45:04.620531 (+ 1425us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:04.620543 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:04.620564 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:45:04.620631 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:04.620635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:04.620776 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:04.620927 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:04.621426 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.622730 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.624553 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.625357 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.625468 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.625655 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:04.627045 (+ 1390us) server_negotiation.cc:299] Negotiation successful
0925 22:45:04.627273 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1318,"thread_start_us":122,"threads_started":1}
I20260925 22:45:04.630199 22190 tablet_service.cc:1467] Processing CreateTablet for tablet 79ac6ff537a34f53b53b7db489d2ecc9 (DEFAULT_TABLE table=test-table [id=c8ef66051b32484c8ce94758ac8ecad6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:04.630198 22321 tablet_service.cc:1467] Processing CreateTablet for tablet 79ac6ff537a34f53b53b7db489d2ecc9 (DEFAULT_TABLE table=test-table [id=c8ef66051b32484c8ce94758ac8ecad6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:04.630203 22452 tablet_service.cc:1467] Processing CreateTablet for tablet 79ac6ff537a34f53b53b7db489d2ecc9 (DEFAULT_TABLE table=test-table [id=c8ef66051b32484c8ce94758ac8ecad6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:04.631446 22452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79ac6ff537a34f53b53b7db489d2ecc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:04.631446 22321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79ac6ff537a34f53b53b7db489d2ecc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:04.631446 22190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79ac6ff537a34f53b53b7db489d2ecc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:04.640122 22537 tablet_bootstrap.cc:492] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a: Bootstrap starting.
I20260925 22:45:04.642328 22537 tablet_bootstrap.cc:654] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:04.643316 22537 log.cc:824] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:04.643415 22535 tablet_bootstrap.cc:492] T 79ac6ff537a34f53b53b7db489d2ecc9 P 32dd31b07bb64da5b1cb07bfa9de518b: Bootstrap starting.
I20260925 22:45:04.643962 22536 tablet_bootstrap.cc:492] T 79ac6ff537a34f53b53b7db489d2ecc9 P d567fe99f22348adb0c0d9833cbdd1ea: Bootstrap starting.
I20260925 22:45:04.645816 22537 tablet_bootstrap.cc:492] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a: No bootstrap required, opened a new log
I20260925 22:45:04.645968 22535 tablet_bootstrap.cc:654] T 79ac6ff537a34f53b53b7db489d2ecc9 P 32dd31b07bb64da5b1cb07bfa9de518b: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:04.646050 22537 ts_tablet_manager.cc:1397] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:45:04.646344 22536 tablet_bootstrap.cc:654] T 79ac6ff537a34f53b53b7db489d2ecc9 P d567fe99f22348adb0c0d9833cbdd1ea: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:04.646970 22535 log.cc:824] T 79ac6ff537a34f53b53b7db489d2ecc9 P 32dd31b07bb64da5b1cb07bfa9de518b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:04.647187 22536 log.cc:824] T 79ac6ff537a34f53b53b7db489d2ecc9 P d567fe99f22348adb0c0d9833cbdd1ea: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:04.649498 22535 tablet_bootstrap.cc:492] T 79ac6ff537a34f53b53b7db489d2ecc9 P 32dd31b07bb64da5b1cb07bfa9de518b: No bootstrap required, opened a new log
I20260925 22:45:04.649731 22535 ts_tablet_manager.cc:1397] T 79ac6ff537a34f53b53b7db489d2ecc9 P 32dd31b07bb64da5b1cb07bfa9de518b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:45:04.650070 22536 tablet_bootstrap.cc:492] T 79ac6ff537a34f53b53b7db489d2ecc9 P d567fe99f22348adb0c0d9833cbdd1ea: No bootstrap required, opened a new log
I20260925 22:45:04.650285 22536 ts_tablet_manager.cc:1397] T 79ac6ff537a34f53b53b7db489d2ecc9 P d567fe99f22348adb0c0d9833cbdd1ea: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 22:45:04.650857 22537 raft_consensus.cc:348] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d567fe99f22348adb0c0d9833cbdd1ea" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37029 } } peers { permanent_uuid: "32dd31b07bb64da5b1cb07bfa9de518b" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37551 } } peers { permanent_uuid: "c2b0fde054e049c999465cef407da18a" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33609 } }
I20260925 22:45:04.651196 22537 raft_consensus.cc:374] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:04.651283 22537 raft_consensus.cc:729] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2b0fde054e049c999465cef407da18a, State: Initialized, Role: FOLLOWER
I20260925 22:45:04.651779 22537 consensus_queue.cc:260] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d567fe99f22348adb0c0d9833cbdd1ea" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37029 } } peers { permanent_uuid: "32dd31b07bb64da5b1cb07bfa9de518b" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37551 } } peers { permanent_uuid: "c2b0fde054e049c999465cef407da18a" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33609 } }
I20260925 22:45:04.652561 22518 heartbeater.cc:503] Master 127.2.74.254:45427 was elected leader, sending a full tablet report...
I20260925 22:45:04.652925 22537 ts_tablet_manager.cc:1428] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 22:45:04.654645 22535 raft_consensus.cc:348] T 79ac6ff537a34f53b53b7db489d2ecc9 P 32dd31b07bb64da5b1cb07bfa9de518b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d567fe99f22348adb0c0d9833cbdd1ea" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37029 } } peers { permanent_uuid: "32dd31b07bb64da5b1cb07bfa9de518b" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37551 } } peers { permanent_uuid: "c2b0fde054e049c999465cef407da18a" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33609 } }
I20260925 22:45:04.654578 22536 raft_consensus.cc:348] T 79ac6ff537a34f53b53b7db489d2ecc9 P d567fe99f22348adb0c0d9833cbdd1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d567fe99f22348adb0c0d9833cbdd1ea" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37029 } } peers { permanent_uuid: "32dd31b07bb64da5b1cb07bfa9de518b" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37551 } } peers { permanent_uuid: "c2b0fde054e049c999465cef407da18a" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33609 } }
I20260925 22:45:04.654902 22535 raft_consensus.cc:374] T 79ac6ff537a34f53b53b7db489d2ecc9 P 32dd31b07bb64da5b1cb07bfa9de518b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:04.654903 22536 raft_consensus.cc:374] T 79ac6ff537a34f53b53b7db489d2ecc9 P d567fe99f22348adb0c0d9833cbdd1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:04.654958 22535 raft_consensus.cc:729] T 79ac6ff537a34f53b53b7db489d2ecc9 P 32dd31b07bb64da5b1cb07bfa9de518b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32dd31b07bb64da5b1cb07bfa9de518b, State: Initialized, Role: FOLLOWER
I20260925 22:45:04.655000 22536 raft_consensus.cc:729] T 79ac6ff537a34f53b53b7db489d2ecc9 P d567fe99f22348adb0c0d9833cbdd1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d567fe99f22348adb0c0d9833cbdd1ea, State: Initialized, Role: FOLLOWER
I20260925 22:45:04.655476 22535 consensus_queue.cc:260] T 79ac6ff537a34f53b53b7db489d2ecc9 P 32dd31b07bb64da5b1cb07bfa9de518b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d567fe99f22348adb0c0d9833cbdd1ea" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37029 } } peers { permanent_uuid: "32dd31b07bb64da5b1cb07bfa9de518b" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37551 } } peers { permanent_uuid: "c2b0fde054e049c999465cef407da18a" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33609 } }
I20260925 22:45:04.655489 22536 consensus_queue.cc:260] T 79ac6ff537a34f53b53b7db489d2ecc9 P d567fe99f22348adb0c0d9833cbdd1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d567fe99f22348adb0c0d9833cbdd1ea" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37029 } } peers { permanent_uuid: "32dd31b07bb64da5b1cb07bfa9de518b" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37551 } } peers { permanent_uuid: "c2b0fde054e049c999465cef407da18a" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33609 } }
I20260925 22:45:04.656510 22536 ts_tablet_manager.cc:1428] T 79ac6ff537a34f53b53b7db489d2ecc9 P d567fe99f22348adb0c0d9833cbdd1ea: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:45:04.656563 22535 ts_tablet_manager.cc:1428] T 79ac6ff537a34f53b53b7db489d2ecc9 P 32dd31b07bb64da5b1cb07bfa9de518b: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:45:04.656921 22387 heartbeater.cc:503] Master 127.2.74.254:45427 was elected leader, sending a full tablet report...
I20260925 22:45:04.657004 22256 heartbeater.cc:503] Master 127.2.74.254:45427 was elected leader, sending a full tablet report...
W20260925 22:45:04.679185 22257 tablet.cc:2369] T 79ac6ff537a34f53b53b7db489d2ecc9 P 32dd31b07bb64da5b1cb07bfa9de518b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:45:04.776907 22541 raft_consensus.cc:482] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:45:04.777166 22541 raft_consensus.cc:504] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d567fe99f22348adb0c0d9833cbdd1ea" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37029 } } peers { permanent_uuid: "32dd31b07bb64da5b1cb07bfa9de518b" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37551 } } peers { permanent_uuid: "c2b0fde054e049c999465cef407da18a" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33609 } }
I20260925 22:45:04.778424 22541 leader_election.cc:288] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d567fe99f22348adb0c0d9833cbdd1ea (127.2.74.194:37029), 32dd31b07bb64da5b1cb07bfa9de518b (127.2.74.193:37551)
I20260925 22:45:04.782912 22531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:04.778887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:58037 (local address 127.2.74.193:37551)
0925 22:45:04.779039 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:04.779044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:04.779385 (+ 341us) server_negotiation.cc:407] Connection header received
0925 22:45:04.779481 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:04.779484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:04.779548 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:04.779627 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:04.779915 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.780534 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.781827 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.782241 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.782390 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.782440 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:04.782692 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 22:45:04.782772 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 22:45:04.782900 22530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:04.778835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.74.195:53069 (local address 127.2.74.194:37029)
0925 22:45:04.779009 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:04.779014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:04.779032 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:45:04.779101 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:04.779104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:04.779166 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:04.779258 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:04.779589 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.780354 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.781755 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.782208 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.782350 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.782415 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:04.782667 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 22:45:04.782749 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:45:04.782912 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:04.778697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.194:37029 (local address 127.2.74.195:53069)
0925 22:45:04.778835 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:04.778852 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:04.778989 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:04.779295 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:04.779297 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:04.779314 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:04.779465 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.779472 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.780476 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.780480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.781603 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.781615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.782378 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.782382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.782490 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.782546 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 22:45:04.782622 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 22:45:04.782732 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":53}
I20260925 22:45:04.782929 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:04.778708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:37551 (local address 127.2.74.195:58037)
0925 22:45:04.779268 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:04.779283 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:04.779389 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:04.779659 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:04.779662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:04.779679 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:04.779802 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.779809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.780650 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.780653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.781664 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.781701 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.782422 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.782428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.782540 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.782558 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 22:45:04.782654 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 22:45:04.782732 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":222,"thread_start_us":134,"threads_started":1}
I20260925 22:45:04.783778 22341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ac6ff537a34f53b53b7db489d2ecc9" candidate_uuid: "c2b0fde054e049c999465cef407da18a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d567fe99f22348adb0c0d9833cbdd1ea" is_pre_election: true
I20260925 22:45:04.783910 22210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ac6ff537a34f53b53b7db489d2ecc9" candidate_uuid: "c2b0fde054e049c999465cef407da18a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32dd31b07bb64da5b1cb07bfa9de518b" is_pre_election: true
I20260925 22:45:04.784157 22210 raft_consensus.cc:2454] T 79ac6ff537a34f53b53b7db489d2ecc9 P 32dd31b07bb64da5b1cb07bfa9de518b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2b0fde054e049c999465cef407da18a in term 0.
I20260925 22:45:04.784157 22341 raft_consensus.cc:2454] T 79ac6ff537a34f53b53b7db489d2ecc9 P d567fe99f22348adb0c0d9833cbdd1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2b0fde054e049c999465cef407da18a in term 0.
I20260925 22:45:04.784718 22405 leader_election.cc:302] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 32dd31b07bb64da5b1cb07bfa9de518b, c2b0fde054e049c999465cef407da18a, d567fe99f22348adb0c0d9833cbdd1ea; no voters:
I20260925 22:45:04.785058 22541 raft_consensus.cc:2781] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:45:04.785168 22541 raft_consensus.cc:482] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:45:04.785234 22541 raft_consensus.cc:3037] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:04.786322 22541 raft_consensus.cc:504] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d567fe99f22348adb0c0d9833cbdd1ea" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37029 } } peers { permanent_uuid: "32dd31b07bb64da5b1cb07bfa9de518b" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37551 } } peers { permanent_uuid: "c2b0fde054e049c999465cef407da18a" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33609 } }
I20260925 22:45:04.786748 22541 leader_election.cc:288] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a [CANDIDATE]: Term 1 election: Requested vote from peers d567fe99f22348adb0c0d9833cbdd1ea (127.2.74.194:37029), 32dd31b07bb64da5b1cb07bfa9de518b (127.2.74.193:37551)
I20260925 22:45:04.787266 22341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ac6ff537a34f53b53b7db489d2ecc9" candidate_uuid: "c2b0fde054e049c999465cef407da18a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d567fe99f22348adb0c0d9833cbdd1ea"
I20260925 22:45:04.787266 22210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ac6ff537a34f53b53b7db489d2ecc9" candidate_uuid: "c2b0fde054e049c999465cef407da18a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32dd31b07bb64da5b1cb07bfa9de518b"
I20260925 22:45:04.787418 22341 raft_consensus.cc:3037] T 79ac6ff537a34f53b53b7db489d2ecc9 P d567fe99f22348adb0c0d9833cbdd1ea [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:04.787418 22210 raft_consensus.cc:3037] T 79ac6ff537a34f53b53b7db489d2ecc9 P 32dd31b07bb64da5b1cb07bfa9de518b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:04.788811 22210 raft_consensus.cc:2454] T 79ac6ff537a34f53b53b7db489d2ecc9 P 32dd31b07bb64da5b1cb07bfa9de518b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2b0fde054e049c999465cef407da18a in term 1.
I20260925 22:45:04.788815 22341 raft_consensus.cc:2454] T 79ac6ff537a34f53b53b7db489d2ecc9 P d567fe99f22348adb0c0d9833cbdd1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2b0fde054e049c999465cef407da18a in term 1.
I20260925 22:45:04.789222 22407 leader_election.cc:302] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 32dd31b07bb64da5b1cb07bfa9de518b, c2b0fde054e049c999465cef407da18a, d567fe99f22348adb0c0d9833cbdd1ea; no voters:
I20260925 22:45:04.789434 22541 raft_consensus.cc:2781] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:04.789692 22541 raft_consensus.cc:686] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a [term 1 LEADER]: Becoming Leader. State: Replica: c2b0fde054e049c999465cef407da18a, State: Running, Role: LEADER
I20260925 22:45:04.790055 22541 consensus_queue.cc:237] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d567fe99f22348adb0c0d9833cbdd1ea" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37029 } } peers { permanent_uuid: "32dd31b07bb64da5b1cb07bfa9de518b" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37551 } } peers { permanent_uuid: "c2b0fde054e049c999465cef407da18a" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33609 } }
I20260925 22:45:04.793727 22068 catalog_manager.cc:5495] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a reported cstate change: term changed from 0 to 1, leader changed from <none> to c2b0fde054e049c999465cef407da18a (127.2.74.195). New cstate: current_term: 1 leader_uuid: "c2b0fde054e049c999465cef407da18a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d567fe99f22348adb0c0d9833cbdd1ea" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32dd31b07bb64da5b1cb07bfa9de518b" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2b0fde054e049c999465cef407da18a" member_type: VOTER last_known_addr { host: "127.2.74.195" port: 33609 } health_report { overall_health: HEALTHY } } }
W20260925 22:45:04.821385 22519 tablet.cc:2369] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:45:04.827121 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:04.823540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33940 (local address 127.2.74.195:33609)
0925 22:45:04.823678 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:04.823681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:04.823696 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:45:04.823773 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:04.823775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:04.823829 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:04.823922 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:45:04.824149 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.824678 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.825507 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.825721 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.825805 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.826066 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:45:04.826622 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:45:04.826755 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:04.826826 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 22:45:04.826878 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 22:45:04.833526 22341 raft_consensus.cc:1261] T 79ac6ff537a34f53b53b7db489d2ecc9 P d567fe99f22348adb0c0d9833cbdd1ea [term 1 FOLLOWER]: Refusing update from remote peer c2b0fde054e049c999465cef407da18a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:04.833545 22210 raft_consensus.cc:1261] T 79ac6ff537a34f53b53b7db489d2ecc9 P 32dd31b07bb64da5b1cb07bfa9de518b [term 1 FOLLOWER]: Refusing update from remote peer c2b0fde054e049c999465cef407da18a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:04.834447 22541 consensus_queue.cc:1035] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32dd31b07bb64da5b1cb07bfa9de518b" member_type: VOTER last_known_addr { host: "127.2.74.193" port: 37551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:04.834756 22545 consensus_queue.cc:1035] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d567fe99f22348adb0c0d9833cbdd1ea" member_type: VOTER last_known_addr { host: "127.2.74.194" port: 37029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:04.842281 22548 mvcc.cc:205] Tried to move back new op lower bound from 7333381344589262848 to 7333381344422891520. Current Snapshot: MvccSnapshot[applied={T|T < 7333381344589262848}]
I20260925 22:45:04.858032 22068 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:45:04.858197 22068 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:45:04.860880 22068 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e750bf1af60e484cac51e9c46882ba67: Sending DeleteTablet for 3 replicas of tablet 79ac6ff537a34f53b53b7db489d2ecc9
I20260925 22:45:04.861915 22321 tablet_service.cc:1514] Processing DeleteTablet for tablet 79ac6ff537a34f53b53b7db489d2ecc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:04 UTC) from {username='slave'} at 127.0.0.1:47076
I20260925 22:45:04.861893 22452 tablet_service.cc:1514] Processing DeleteTablet for tablet 79ac6ff537a34f53b53b7db489d2ecc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:04 UTC) from {username='slave'} at 127.0.0.1:33936
I20260925 22:45:04.862474 22558 tablet_replica.cc:330] T 79ac6ff537a34f53b53b7db489d2ecc9 P d567fe99f22348adb0c0d9833cbdd1ea: stopping tablet replica
I20260925 22:45:04.862509 22190 tablet_service.cc:1514] Processing DeleteTablet for tablet 79ac6ff537a34f53b53b7db489d2ecc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:04 UTC) from {username='slave'} at 127.0.0.1:39704
I20260925 22:45:04.862823 22559 tablet_replica.cc:330] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a: stopping tablet replica
I20260925 22:45:04.863138 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 22128
I20260925 22:45:04.863963 22558 raft_consensus.cc:2229] T 79ac6ff537a34f53b53b7db489d2ecc9 P d567fe99f22348adb0c0d9833cbdd1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:45:04.864336 22558 raft_consensus.cc:2258] T 79ac6ff537a34f53b53b7db489d2ecc9 P d567fe99f22348adb0c0d9833cbdd1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:45:04.864725 22559 raft_consensus.cc:2229] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:45:04.865126 22559 raft_consensus.cc:2258] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:45:04.866454 22559 ts_tablet_manager.cc:1905] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:45:04.867604 22558 ts_tablet_manager.cc:1905] T 79ac6ff537a34f53b53b7db489d2ecc9 P d567fe99f22348adb0c0d9833cbdd1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:45:04.869460 22559 ts_tablet_manager.cc:1918] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:45:04.869582 22559 log.cc:1196] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-2/wal/wals/79ac6ff537a34f53b53b7db489d2ecc9
I20260925 22:45:04.869920 22559 ts_tablet_manager.cc:1939] T 79ac6ff537a34f53b53b7db489d2ecc9 P c2b0fde054e049c999465cef407da18a: Deleting consensus metadata
I20260925 22:45:04.870572 22558 ts_tablet_manager.cc:1918] T 79ac6ff537a34f53b53b7db489d2ecc9 P d567fe99f22348adb0c0d9833cbdd1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:45:04.870683 22558 log.cc:1196] T 79ac6ff537a34f53b53b7db489d2ecc9 P d567fe99f22348adb0c0d9833cbdd1ea: Deleting WAL directory at /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376047580784-2347-0/minicluster-data/ts-1/wal/wals/79ac6ff537a34f53b53b7db489d2ecc9
I20260925 22:45:04.871047 22558 ts_tablet_manager.cc:1939] T 79ac6ff537a34f53b53b7db489d2ecc9 P d567fe99f22348adb0c0d9833cbdd1ea: Deleting consensus metadata
I20260925 22:45:04.871470 22055 catalog_manager.cc:4841] TS c2b0fde054e049c999465cef407da18a (127.2.74.195:33609): tablet 79ac6ff537a34f53b53b7db489d2ecc9 (table test-table [id=c8ef66051b32484c8ce94758ac8ecad6]) successfully deleted
I20260925 22:45:04.872272 22054 catalog_manager.cc:4841] TS d567fe99f22348adb0c0d9833cbdd1ea (127.2.74.194:37029): tablet 79ac6ff537a34f53b53b7db489d2ecc9 (table test-table [id=c8ef66051b32484c8ce94758ac8ecad6]) successfully deleted
W20260925 22:45:04.876581 22052 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.74.193:37551 (error 108)
I20260925 22:45:04.876632 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 22259
I20260925 22:45:04.878295 22534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:04.877782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.193:37551 (local address 127.0.0.1:39716)
0925 22:45:04.878135 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:04.878199 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.193:37551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274}
W20260925 22:45:04.878822 22052 catalog_manager.cc:4584] TS 32dd31b07bb64da5b1cb07bfa9de518b (127.2.74.193:37551): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 79ac6ff537a34f53b53b7db489d2ecc9: Network error: Client connection negotiation failed: client connection to 127.2.74.193:37551: connect: Connection refused (error 111)
I20260925 22:45:04.882678 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 22390
I20260925 22:45:04.888849 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 22037
2026-09-25T22:45:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1217 ms)
[----------] 6 tests from EncryptionPolicyTest (11204 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:04 dist-test-slave-csnl krb5kdc[22564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:04 dist-test-slave-csnl krb5kdc[22564](info): set up 2 sockets
Sep 25 22:45:04 dist-test-slave-csnl krb5kdc[22564](info): commencing operation
krb5kdc: starting...
W20260925 22:45:05.931373 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:45:06 dist-test-slave-csnl krb5kdc[22564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:45:06.062188 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33825
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:06.179471 22577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:06.179852 22577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:06.179961 22577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:06.184469 22577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:06.184588 22577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:06.184626 22577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:06.184686 22577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:06.184744 22577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:06.190856 22577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33825
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:06.192147 22577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:06.193250 22577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:06.202425 22582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:06.202425 22585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:06.202445 22583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:06.202734 22577 server_base.cc:1034] running on GCE node
I20260925 22:45:06.203433 22577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:06.204691 22577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:06.205868 22577 hybrid_clock.cc:648] HybridClock initialized: now 1790376306205852 us; error 30 us; skew 500 ppm
Sep 25 22:45:06 dist-test-slave-csnl krb5kdc[22564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:06.214757 22577 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:45:06.215848 22577 webserver.cc:460] Webserver started at http://127.0.0.1:46329/ using document root <none> and password file <none>
I20260925 22:45:06.216434 22577 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:06.216497 22577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:06.216665 22577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:06.218869 22577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "2dd21554f62948fa9ea392c54c9a77ed"
format_stamp: "Formatted at 2026-09-25 22:45:06 on dist-test-slave-csnl"
server_key: "066fc281b51c38ec29d0d1b9644648e6"
server_key_iv: "a72e791b83431a9424e4533fdf0a70c2"
server_key_version: "encryptionkey@0"
I20260925 22:45:06.219537 22577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "2dd21554f62948fa9ea392c54c9a77ed"
format_stamp: "Formatted at 2026-09-25 22:45:06 on dist-test-slave-csnl"
server_key: "066fc281b51c38ec29d0d1b9644648e6"
server_key_iv: "a72e791b83431a9424e4533fdf0a70c2"
server_key_version: "encryptionkey@0"
I20260925 22:45:06.223727 22577 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:45:06.226687 22592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:06.228229 22577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 22:45:06.228440 22577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "2dd21554f62948fa9ea392c54c9a77ed"
format_stamp: "Formatted at 2026-09-25 22:45:06 on dist-test-slave-csnl"
server_key: "066fc281b51c38ec29d0d1b9644648e6"
server_key_iv: "a72e791b83431a9424e4533fdf0a70c2"
server_key_version: "encryptionkey@0"
I20260925 22:45:06.228590 22577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:06.249493 22577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:06.252846 22577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:06.253077 22577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:06.260881 22577 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33825
I20260925 22:45:06.260912 22644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33825 every 8 connection(s)
I20260925 22:45:06.261986 22577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:06.265029 22645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:06.269657 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 22577
I20260925 22:45:06.269825 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:45:06.270112 2347 external_mini_cluster.cc:1428] Setting key 2c45e8ab9f3612c603fafb934e6c62cc
I20260925 22:45:06.271992 22645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dd21554f62948fa9ea392c54c9a77ed: Bootstrap starting.
I20260925 22:45:06.274359 22645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dd21554f62948fa9ea392c54c9a77ed: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:06.275244 22645 log.cc:824] T 00000000000000000000000000000000 P 2dd21554f62948fa9ea392c54c9a77ed: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:06.277292 22645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dd21554f62948fa9ea392c54c9a77ed: No bootstrap required, opened a new log
Sep 25 22:45:06 dist-test-slave-csnl krb5kdc[22564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 22:45:06.281558 22645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dd21554f62948fa9ea392c54c9a77ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd21554f62948fa9ea392c54c9a77ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33825 } }
I20260925 22:45:06.281857 22645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2dd21554f62948fa9ea392c54c9a77ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:06.281942 22645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dd21554f62948fa9ea392c54c9a77ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dd21554f62948fa9ea392c54c9a77ed, State: Initialized, Role: FOLLOWER
I20260925 22:45:06.282491 22645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dd21554f62948fa9ea392c54c9a77ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd21554f62948fa9ea392c54c9a77ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33825 } }
I20260925 22:45:06.282660 22645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dd21554f62948fa9ea392c54c9a77ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:06.282756 22645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dd21554f62948fa9ea392c54c9a77ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:06.282883 22645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dd21554f62948fa9ea392c54c9a77ed [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:06.283875 22645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dd21554f62948fa9ea392c54c9a77ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd21554f62948fa9ea392c54c9a77ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33825 } }
I20260925 22:45:06.284250 22645 leader_election.cc:302] T 00000000000000000000000000000000 P 2dd21554f62948fa9ea392c54c9a77ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dd21554f62948fa9ea392c54c9a77ed; no voters:
I20260925 22:45:06.284632 22645 leader_election.cc:288] T 00000000000000000000000000000000 P 2dd21554f62948fa9ea392c54c9a77ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:06.285028 22651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dd21554f62948fa9ea392c54c9a77ed [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:06.285223 22651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dd21554f62948fa9ea392c54c9a77ed [term 1 LEADER]: Becoming Leader. State: Replica: 2dd21554f62948fa9ea392c54c9a77ed, State: Running, Role: LEADER
I20260925 22:45:06.285539 22651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dd21554f62948fa9ea392c54c9a77ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd21554f62948fa9ea392c54c9a77ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33825 } }
I20260925 22:45:06.285697 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:06.271760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37677 (local address 127.0.0.1:33825)
0925 22:45:06.272287 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:06.272304 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:06.272350 (+ 46us) server_negotiation.cc:407] Connection header received
0925 22:45:06.273042 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:06.273058 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:06.273437 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:06.273807 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:06.274721 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:06.276014 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:06.276922 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:06.277443 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:06.277610 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:06.280413 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:06.280420 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:06.280437 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:06.280470 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:06.282345 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:06.283016 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:06.283022 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:06.283026 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:06.283141 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:06.283491 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:06.283501 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:06.283503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:06.283911 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:06.284141 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:06.284452 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 22:45:06.284711 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":154,"threads_started":1}
I20260925 22:45:06.286028 22645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dd21554f62948fa9ea392c54c9a77ed [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:06.286772 22651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dd21554f62948fa9ea392c54c9a77ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dd21554f62948fa9ea392c54c9a77ed. Latest consensus state: current_term: 1 leader_uuid: "2dd21554f62948fa9ea392c54c9a77ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd21554f62948fa9ea392c54c9a77ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33825 } } }
I20260925 22:45:06.286908 22651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dd21554f62948fa9ea392c54c9a77ed [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:06.287207 22652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dd21554f62948fa9ea392c54c9a77ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2dd21554f62948fa9ea392c54c9a77ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd21554f62948fa9ea392c54c9a77ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33825 } } }
I20260925 22:45:06.287298 22652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dd21554f62948fa9ea392c54c9a77ed [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:06.287344 22658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:06.289137 2347 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:45:06.290663 22658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:06.297324 22658 catalog_manager.cc:1347] Generated new cluster ID: 00a1ab7a71df43c7b079443f76e00510
I20260925 22:45:06.297401 22658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:06.301426 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:06.291739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54399 (local address 127.0.0.1:33825)
0925 22:45:06.291861 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:06.291865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:06.292177 (+ 312us) server_negotiation.cc:407] Connection header received
0925 22:45:06.292382 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:06.292386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:06.292439 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:06.292527 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:06.293354 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:06.294127 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:06.295289 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:06.295654 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:06.295764 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:06.297638 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:06.297642 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:06.297645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:06.297675 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:06.299499 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:06.300054 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:06.300057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:06.300059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:06.300141 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:06.300477 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:06.300481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:06.300482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:06.300677 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:06.300781 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:06.301160 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 22:45:06.301264 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 22:45:06.318573 22658 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:06.319749 22658 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:06.337668 22658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2dd21554f62948fa9ea392c54c9a77ed: Generated new TSK 0
I20260925 22:45:06.338655 22658 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:45:06.354149 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 22577
W20260925 22:45:06.360015 22562 connection.cc:535] client connection to 127.0.0.1:33825 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33825): Connection reset by peer (error 104)
2026-09-25T22:45:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1463 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:06 dist-test-slave-csnl krb5kdc[22676](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:06 dist-test-slave-csnl krb5kdc[22676](info): set up 2 sockets
Sep 25 22:45:06 dist-test-slave-csnl krb5kdc[22676](info): commencing operation
krb5kdc: starting...
W20260925 22:45:07.395207 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:45:07 dist-test-slave-csnl krb5kdc[22676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:45:07.521634 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39229
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:07.638481 22689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:07.638796 22689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:07.638840 22689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:07.643163 22689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:07.643251 22689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:07.643273 22689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:07.643291 22689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:07.643363 22689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:07.649076 22689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39229
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:07.650292 22689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:07.651439 22689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:45:07.660413 22689 server_base.cc:1034] running on GCE node
W20260925 22:45:07.660353 22695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:07.660360 22697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:07.660353 22694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:07.661253 22689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:07.662499 22689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:07.663704 22689 hybrid_clock.cc:648] HybridClock initialized: now 1790376307663687 us; error 33 us; skew 500 ppm
Sep 25 22:45:07 dist-test-slave-csnl krb5kdc[22676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:07.666616 22689 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:45:07.667613 22689 webserver.cc:460] Webserver started at http://127.0.0.1:44535/ using document root <none> and password file <none>
I20260925 22:45:07.668174 22689 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:07.668242 22689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:07.668414 22689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:07.670610 22689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "0bec53f307ad42b4b32ac17bc832e8f3"
format_stamp: "Formatted at 2026-09-25 22:45:07 on dist-test-slave-csnl"
server_key: "a98fea1a5bf5554b652436eae7578060"
server_key_iv: "450d955206fa79de29a92c698cf1b3a4"
server_key_version: "encryptionkey@0"
I20260925 22:45:07.671275 22689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "0bec53f307ad42b4b32ac17bc832e8f3"
format_stamp: "Formatted at 2026-09-25 22:45:07 on dist-test-slave-csnl"
server_key: "a98fea1a5bf5554b652436eae7578060"
server_key_iv: "450d955206fa79de29a92c698cf1b3a4"
server_key_version: "encryptionkey@0"
I20260925 22:45:07.675607 22689 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:45:07.678453 22704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:07.679996 22689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:07.680140 22689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "0bec53f307ad42b4b32ac17bc832e8f3"
format_stamp: "Formatted at 2026-09-25 22:45:07 on dist-test-slave-csnl"
server_key: "a98fea1a5bf5554b652436eae7578060"
server_key_iv: "450d955206fa79de29a92c698cf1b3a4"
server_key_version: "encryptionkey@0"
I20260925 22:45:07.680291 22689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:07.697413 22689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:07.701041 22689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:07.701229 22689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:07.708631 22689 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39229
I20260925 22:45:07.708652 22756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39229 every 8 connection(s)
I20260925 22:45:07.709717 22689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:07.712821 22757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:07.719002 22757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bec53f307ad42b4b32ac17bc832e8f3: Bootstrap starting.
I20260925 22:45:07.719020 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 22689
I20260925 22:45:07.719223 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:45:07.719561 2347 external_mini_cluster.cc:1428] Setting key 83a5c03071df7f614f0e1cc0cd7daa4a
I20260925 22:45:07.721889 22757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bec53f307ad42b4b32ac17bc832e8f3: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:07.722736 22757 log.cc:824] T 00000000000000000000000000000000 P 0bec53f307ad42b4b32ac17bc832e8f3: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:07.724898 22757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bec53f307ad42b4b32ac17bc832e8f3: No bootstrap required, opened a new log
I20260925 22:45:07.728822 22757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bec53f307ad42b4b32ac17bc832e8f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bec53f307ad42b4b32ac17bc832e8f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39229 } }
I20260925 22:45:07.729153 22757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bec53f307ad42b4b32ac17bc832e8f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:07.729239 22757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bec53f307ad42b4b32ac17bc832e8f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bec53f307ad42b4b32ac17bc832e8f3, State: Initialized, Role: FOLLOWER
I20260925 22:45:07.729606 22760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:07.721074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51725 (local address 127.0.0.1:39229)
0925 22:45:07.721587 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:07.721604 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:07.721643 (+ 39us) server_negotiation.cc:407] Connection header received
0925 22:45:07.722472 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:07.722493 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:07.722941 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:07.723350 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:07.723804 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:07.724801 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:07.725809 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:07.726328 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:07.726510 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:07.727544 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:07.727550 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:07.727567 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:07.728006 (+ 439us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:07.728067 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:07.728092 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:07.728257 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:07.728416 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 22:45:07.728580 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":146,"threads_started":1}
I20260925 22:45:07.729784 22757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bec53f307ad42b4b32ac17bc832e8f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bec53f307ad42b4b32ac17bc832e8f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39229 } }
I20260925 22:45:07.729984 22757 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bec53f307ad42b4b32ac17bc832e8f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:07.730077 22757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bec53f307ad42b4b32ac17bc832e8f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:07.730187 22757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bec53f307ad42b4b32ac17bc832e8f3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:07.731487 22757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bec53f307ad42b4b32ac17bc832e8f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bec53f307ad42b4b32ac17bc832e8f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39229 } }
I20260925 22:45:07.731920 22757 leader_election.cc:302] T 00000000000000000000000000000000 P 0bec53f307ad42b4b32ac17bc832e8f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bec53f307ad42b4b32ac17bc832e8f3; no voters:
I20260925 22:45:07.732327 22757 leader_election.cc:288] T 00000000000000000000000000000000 P 0bec53f307ad42b4b32ac17bc832e8f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:07.732398 22762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bec53f307ad42b4b32ac17bc832e8f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:07.732616 22762 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bec53f307ad42b4b32ac17bc832e8f3 [term 1 LEADER]: Becoming Leader. State: Replica: 0bec53f307ad42b4b32ac17bc832e8f3, State: Running, Role: LEADER
I20260925 22:45:07.733304 2347 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:45:07.733366 22762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bec53f307ad42b4b32ac17bc832e8f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bec53f307ad42b4b32ac17bc832e8f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39229 } }
I20260925 22:45:07.733757 22757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bec53f307ad42b4b32ac17bc832e8f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:07.735309 22763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bec53f307ad42b4b32ac17bc832e8f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bec53f307ad42b4b32ac17bc832e8f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bec53f307ad42b4b32ac17bc832e8f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39229 } } }
I20260925 22:45:07.735476 22763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bec53f307ad42b4b32ac17bc832e8f3 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:07.735832 22764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bec53f307ad42b4b32ac17bc832e8f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bec53f307ad42b4b32ac17bc832e8f3. Latest consensus state: current_term: 1 leader_uuid: "0bec53f307ad42b4b32ac17bc832e8f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bec53f307ad42b4b32ac17bc832e8f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39229 } } }
I20260925 22:45:07.735939 22764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bec53f307ad42b4b32ac17bc832e8f3 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:07.736480 22778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:07.740177 22778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:07.740799 22760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:07.735766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54593 (local address 127.0.0.1:39229)
0925 22:45:07.735906 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:07.735911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:07.736215 (+ 304us) server_negotiation.cc:407] Connection header received
0925 22:45:07.736416 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:07.736420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:07.736475 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:07.736582 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:07.736869 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:07.737607 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:07.738796 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:07.739189 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:07.739299 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:07.740262 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:07.740266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:07.740269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:07.740284 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:07.740315 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:07.740328 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:07.740424 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:07.740609 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 22:45:07.740650 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:45:07.746438 22778 catalog_manager.cc:1347] Generated new cluster ID: 61213f7b51e34f109a7625e779c0b7c7
I20260925 22:45:07.746533 22778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:07.775393 22778 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:07.776799 22778 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:07.785949 22778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bec53f307ad42b4b32ac17bc832e8f3: Generated new TSK 0
I20260925 22:45:07.786780 22778 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:45:07.798759 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 22689
W20260925 22:45:07.804399 22674 connection.cc:535] client connection to 127.0.0.1:39229 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39229): Connection reset by peer (error 104)
2026-09-25T22:45:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1444 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:07 dist-test-slave-csnl krb5kdc[22788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:07 dist-test-slave-csnl krb5kdc[22788](info): set up 2 sockets
Sep 25 22:45:07 dist-test-slave-csnl krb5kdc[22788](info): commencing operation
krb5kdc: starting...
W20260925 22:45:08.836447 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:45:08 dist-test-slave-csnl krb5kdc[22788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:45:08.962093 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42193
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:09.078979 22801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:09.079309 22801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:09.079391 22801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:09.083613 22801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:09.083709 22801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:09.083762 22801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:09.083818 22801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:09.083859 22801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:09.089285 22801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42193
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:09.090420 22801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:09.091519 22801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:09.100899 22806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:09.100888 22807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:09.100870 22809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:09.101203 22801 server_base.cc:1034] running on GCE node
I20260925 22:45:09.101814 22801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:09.103135 22801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:09.104350 22801 hybrid_clock.cc:648] HybridClock initialized: now 1790376309104332 us; error 35 us; skew 500 ppm
Sep 25 22:45:09 dist-test-slave-csnl krb5kdc[22788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:09.107236 22801 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:45:09.108166 22801 webserver.cc:460] Webserver started at http://127.0.0.1:43893/ using document root <none> and password file <none>
I20260925 22:45:09.108763 22801 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:09.108860 22801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:09.109100 22801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:09.111333 22801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "d09bfeff72e54acca9f57ea3948793ff"
format_stamp: "Formatted at 2026-09-25 22:45:09 on dist-test-slave-csnl"
server_key: "30665be7c869e1ecf48c41e41ff6a8cc"
server_key_iv: "3dbf4b2a5cc1ee939d7448944ac76078"
server_key_version: "encryptionkey@0"
I20260925 22:45:09.111937 22801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "d09bfeff72e54acca9f57ea3948793ff"
format_stamp: "Formatted at 2026-09-25 22:45:09 on dist-test-slave-csnl"
server_key: "30665be7c869e1ecf48c41e41ff6a8cc"
server_key_iv: "3dbf4b2a5cc1ee939d7448944ac76078"
server_key_version: "encryptionkey@0"
I20260925 22:45:09.115900 22801 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:45:09.118553 22816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:09.119967 22801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:09.120131 22801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "d09bfeff72e54acca9f57ea3948793ff"
format_stamp: "Formatted at 2026-09-25 22:45:09 on dist-test-slave-csnl"
server_key: "30665be7c869e1ecf48c41e41ff6a8cc"
server_key_iv: "3dbf4b2a5cc1ee939d7448944ac76078"
server_key_version: "encryptionkey@0"
I20260925 22:45:09.120268 22801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:09.125124 22801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:09.128172 22801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:09.128381 22801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:09.135572 22801 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42193
I20260925 22:45:09.135632 22868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42193 every 8 connection(s)
I20260925 22:45:09.136600 22801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:09.138912 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 22801
I20260925 22:45:09.139127 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:45:09.139439 2347 external_mini_cluster.cc:1428] Setting key 1a4c71cde243cbc6dea66bce35dc82e6
I20260925 22:45:09.139845 22869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:09.147066 22869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d09bfeff72e54acca9f57ea3948793ff: Bootstrap starting.
I20260925 22:45:09.149850 22869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d09bfeff72e54acca9f57ea3948793ff: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:09.150586 22869 log.cc:824] T 00000000000000000000000000000000 P d09bfeff72e54acca9f57ea3948793ff: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:09.152707 22869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d09bfeff72e54acca9f57ea3948793ff: No bootstrap required, opened a new log
I20260925 22:45:09.156347 22869 raft_consensus.cc:348] T 00000000000000000000000000000000 P d09bfeff72e54acca9f57ea3948793ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09bfeff72e54acca9f57ea3948793ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42193 } }
I20260925 22:45:09.156629 22869 raft_consensus.cc:374] T 00000000000000000000000000000000 P d09bfeff72e54acca9f57ea3948793ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:09.156698 22869 raft_consensus.cc:729] T 00000000000000000000000000000000 P d09bfeff72e54acca9f57ea3948793ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09bfeff72e54acca9f57ea3948793ff, State: Initialized, Role: FOLLOWER
I20260925 22:45:09.157239 22869 consensus_queue.cc:260] T 00000000000000000000000000000000 P d09bfeff72e54acca9f57ea3948793ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09bfeff72e54acca9f57ea3948793ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42193 } }
I20260925 22:45:09.157405 22869 raft_consensus.cc:388] T 00000000000000000000000000000000 P d09bfeff72e54acca9f57ea3948793ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:09.157476 22869 raft_consensus.cc:482] T 00000000000000000000000000000000 P d09bfeff72e54acca9f57ea3948793ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:09.157569 22869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d09bfeff72e54acca9f57ea3948793ff [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:09.158432 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:09.140914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56497 (local address 127.0.0.1:42193)
0925 22:45:09.141411 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:09.141433 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:09.141477 (+ 44us) server_negotiation.cc:407] Connection header received
0925 22:45:09.142317 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:09.142343 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:09.142767 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:09.143194 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:09.156757 (+ 13563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:09.156765 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:09.156783 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:09.156802 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:09.156884 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:09.156917 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:09.157038 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:09.157217 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 22:45:09.157391 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":135,"threads_started":1}
I20260925 22:45:09.158836 22869 raft_consensus.cc:504] T 00000000000000000000000000000000 P d09bfeff72e54acca9f57ea3948793ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09bfeff72e54acca9f57ea3948793ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42193 } }
I20260925 22:45:09.159238 22869 leader_election.cc:302] T 00000000000000000000000000000000 P d09bfeff72e54acca9f57ea3948793ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d09bfeff72e54acca9f57ea3948793ff; no voters:
I20260925 22:45:09.159632 22869 leader_election.cc:288] T 00000000000000000000000000000000 P d09bfeff72e54acca9f57ea3948793ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:09.159755 22874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d09bfeff72e54acca9f57ea3948793ff [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:09.159996 22874 raft_consensus.cc:686] T 00000000000000000000000000000000 P d09bfeff72e54acca9f57ea3948793ff [term 1 LEADER]: Becoming Leader. State: Replica: d09bfeff72e54acca9f57ea3948793ff, State: Running, Role: LEADER
I20260925 22:45:09.160375 22874 consensus_queue.cc:237] T 00000000000000000000000000000000 P d09bfeff72e54acca9f57ea3948793ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09bfeff72e54acca9f57ea3948793ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42193 } }
I20260925 22:45:09.161084 22869 sys_catalog.cc:564] T 00000000000000000000000000000000 P d09bfeff72e54acca9f57ea3948793ff [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:09.161893 2347 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:45:09.162237 22875 sys_catalog.cc:455] T 00000000000000000000000000000000 P d09bfeff72e54acca9f57ea3948793ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d09bfeff72e54acca9f57ea3948793ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09bfeff72e54acca9f57ea3948793ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42193 } } }
I20260925 22:45:09.162374 22875 sys_catalog.cc:458] T 00000000000000000000000000000000 P d09bfeff72e54acca9f57ea3948793ff [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:09.162788 22883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:09.164132 22876 sys_catalog.cc:455] T 00000000000000000000000000000000 P d09bfeff72e54acca9f57ea3948793ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader d09bfeff72e54acca9f57ea3948793ff. Latest consensus state: current_term: 1 leader_uuid: "d09bfeff72e54acca9f57ea3948793ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09bfeff72e54acca9f57ea3948793ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42193 } } }
I20260925 22:45:09.164269 22876 sys_catalog.cc:458] T 00000000000000000000000000000000 P d09bfeff72e54acca9f57ea3948793ff [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:09.166913 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:09.164858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47665 (local address 127.0.0.1:42193)
0925 22:45:09.164981 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:09.164985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:09.165264 (+ 279us) server_negotiation.cc:407] Connection header received
0925 22:45:09.165470 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:09.165474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:09.165522 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:09.165624 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:09.166458 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:09.166462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:09.166464 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:09.166478 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:09.166513 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:09.166527 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:09.166624 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:09.166739 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 22:45:09.166781 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 22:45:09.166975 22883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:09.173996 22883 catalog_manager.cc:1347] Generated new cluster ID: 6b200daf15eb405f9072bb3a500fe957
I20260925 22:45:09.174100 22883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:09.203457 22883 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:09.203881 22883 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:09.213120 22883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d09bfeff72e54acca9f57ea3948793ff: Generated new TSK 0
I20260925 22:45:09.213938 22883 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:45:09.220527 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 22801
2026-09-25T22:45:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:09 dist-test-slave-csnl krb5kdc[22900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:09 dist-test-slave-csnl krb5kdc[22900](info): set up 2 sockets
Sep 25 22:45:09 dist-test-slave-csnl krb5kdc[22900](info): commencing operation
krb5kdc: starting...
W20260925 22:45:10.279307 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:45:10 dist-test-slave-csnl krb5kdc[22900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:45:10.406038 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40351
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:10.520787 22913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:10.521098 22913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:10.521144 22913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:10.525599 22913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:10.525701 22913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:10.525723 22913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:10.525741 22913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:10.525758 22913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:10.531895 22913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:10.533118 22913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:10.534230 22913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:10.543787 22919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:10.543787 22921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:10.543844 22913 server_base.cc:1034] running on GCE node
W20260925 22:45:10.543789 22918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:10.544713 22913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:10.545977 22913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:10.547188 22913 hybrid_clock.cc:648] HybridClock initialized: now 1790376310547157 us; error 51 us; skew 500 ppm
Sep 25 22:45:10 dist-test-slave-csnl krb5kdc[22900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:10.549998 22913 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:45:10.551095 22913 webserver.cc:460] Webserver started at http://127.0.0.1:37047/ using document root <none> and password file <none>
I20260925 22:45:10.551700 22913 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:10.551805 22913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:10.552053 22913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:10.554301 22913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "305428cf5dbd4d2a8f2050b28184211d"
format_stamp: "Formatted at 2026-09-25 22:45:10 on dist-test-slave-csnl"
server_key: "7a28d25a0b593521b53b69f650819eaf"
server_key_iv: "7aaf34d0428d48388226af16076ff409"
server_key_version: "encryptionkey@0"
I20260925 22:45:10.555043 22913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "305428cf5dbd4d2a8f2050b28184211d"
format_stamp: "Formatted at 2026-09-25 22:45:10 on dist-test-slave-csnl"
server_key: "7a28d25a0b593521b53b69f650819eaf"
server_key_iv: "7aaf34d0428d48388226af16076ff409"
server_key_version: "encryptionkey@0"
I20260925 22:45:10.559087 22913 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:45:10.561842 22928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:10.563515 22913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:10.563725 22913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "305428cf5dbd4d2a8f2050b28184211d"
format_stamp: "Formatted at 2026-09-25 22:45:10 on dist-test-slave-csnl"
server_key: "7a28d25a0b593521b53b69f650819eaf"
server_key_iv: "7aaf34d0428d48388226af16076ff409"
server_key_version: "encryptionkey@0"
I20260925 22:45:10.563884 22913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:10.577397 22913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:10.581120 22913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:10.581388 22913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:10.589380 22913 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40351
I20260925 22:45:10.589401 22980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40351 every 8 connection(s)
I20260925 22:45:10.590518 22913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:10.592847 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 22913
I20260925 22:45:10.593000 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:45:10.593333 2347 external_mini_cluster.cc:1428] Setting key 5002f87021731f0b9f1143dc7aabb485
I20260925 22:45:10.593924 22981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:10.601866 22981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 305428cf5dbd4d2a8f2050b28184211d: Bootstrap starting.
Sep 25 22:45:10 dist-test-slave-csnl krb5kdc[22900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 22:45:10.604895 22981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 305428cf5dbd4d2a8f2050b28184211d: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:10.605824 22981 log.cc:824] T 00000000000000000000000000000000 P 305428cf5dbd4d2a8f2050b28184211d: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:10.608070 22981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 305428cf5dbd4d2a8f2050b28184211d: No bootstrap required, opened a new log
I20260925 22:45:10.609673 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:10.594921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51305 (local address 127.0.0.1:40351)
0925 22:45:10.595480 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:10.595508 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:10.595548 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:45:10.596234 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:10.596253 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:10.596626 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:10.596937 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:10.598102 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:10.599244 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:10.600474 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:10.601022 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:10.601164 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:10.604170 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:10.604176 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:10.604193 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:10.604230 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:10.606372 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:10.606934 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:10.606940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:10.606997 (+ 57us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:10.607176 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:10.607524 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:10.607528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:10.607534 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:10.607963 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:10.608197 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:10.608548 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 22:45:10.608878 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":173,"threads_started":1}
I20260925 22:45:10.612624 22981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 305428cf5dbd4d2a8f2050b28184211d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "305428cf5dbd4d2a8f2050b28184211d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40351 } }
I20260925 22:45:10.612972 22981 raft_consensus.cc:374] T 00000000000000000000000000000000 P 305428cf5dbd4d2a8f2050b28184211d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:10.613035 22981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 305428cf5dbd4d2a8f2050b28184211d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 305428cf5dbd4d2a8f2050b28184211d, State: Initialized, Role: FOLLOWER
I20260925 22:45:10.613335 2347 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:45:10.613548 22981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 305428cf5dbd4d2a8f2050b28184211d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "305428cf5dbd4d2a8f2050b28184211d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40351 } }
I20260925 22:45:10.613708 22981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 305428cf5dbd4d2a8f2050b28184211d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:10.613781 22981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 305428cf5dbd4d2a8f2050b28184211d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:10.613919 22981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 305428cf5dbd4d2a8f2050b28184211d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:10.615150 22981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 305428cf5dbd4d2a8f2050b28184211d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "305428cf5dbd4d2a8f2050b28184211d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40351 } }
I20260925 22:45:10.615588 22981 leader_election.cc:302] T 00000000000000000000000000000000 P 305428cf5dbd4d2a8f2050b28184211d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 305428cf5dbd4d2a8f2050b28184211d; no voters:
I20260925 22:45:10.615983 22981 leader_election.cc:288] T 00000000000000000000000000000000 P 305428cf5dbd4d2a8f2050b28184211d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:10.616117 22991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 305428cf5dbd4d2a8f2050b28184211d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:10.616362 22991 raft_consensus.cc:686] T 00000000000000000000000000000000 P 305428cf5dbd4d2a8f2050b28184211d [term 1 LEADER]: Becoming Leader. State: Replica: 305428cf5dbd4d2a8f2050b28184211d, State: Running, Role: LEADER
I20260925 22:45:10.616811 22991 consensus_queue.cc:237] T 00000000000000000000000000000000 P 305428cf5dbd4d2a8f2050b28184211d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "305428cf5dbd4d2a8f2050b28184211d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40351 } }
I20260925 22:45:10.617525 22981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 305428cf5dbd4d2a8f2050b28184211d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:10.618255 22993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 305428cf5dbd4d2a8f2050b28184211d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 305428cf5dbd4d2a8f2050b28184211d. Latest consensus state: current_term: 1 leader_uuid: "305428cf5dbd4d2a8f2050b28184211d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "305428cf5dbd4d2a8f2050b28184211d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40351 } } }
I20260925 22:45:10.618394 22993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 305428cf5dbd4d2a8f2050b28184211d [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:10.619102 22992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 305428cf5dbd4d2a8f2050b28184211d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "305428cf5dbd4d2a8f2050b28184211d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "305428cf5dbd4d2a8f2050b28184211d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40351 } } }
I20260925 22:45:10.619210 22992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 305428cf5dbd4d2a8f2050b28184211d [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:10.619606 22998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:10.624123 22998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:10.625340 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:10.615773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39423 (local address 127.0.0.1:40351)
0925 22:45:10.615911 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:10.615916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:10.616076 (+ 160us) server_negotiation.cc:407] Connection header received
0925 22:45:10.616214 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:10.616217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:10.616274 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:10.616367 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:10.617369 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:10.618252 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:10.619615 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:10.620014 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:10.620138 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:10.621874 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:10.621877 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:10.621880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:10.621905 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:10.623511 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:10.624009 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:10.624013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:10.624015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:10.624107 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:10.624429 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:10.624435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:10.624439 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:10.624642 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:10.624793 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:10.625048 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 22:45:10.625175 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:45:10.630915 22998 catalog_manager.cc:1347] Generated new cluster ID: 08da12fa23fa46ad950380c0b5daa7d3
I20260925 22:45:10.631079 22998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:10.650174 22998 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:10.651587 22998 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:10.660688 22998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 305428cf5dbd4d2a8f2050b28184211d: Generated new TSK 0
I20260925 22:45:10.661640 22998 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:45:10.670711 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 22913
2026-09-25T22:45:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1449 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:10 dist-test-slave-csnl krb5kdc[23012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:10 dist-test-slave-csnl krb5kdc[23012](info): set up 2 sockets
Sep 25 22:45:10 dist-test-slave-csnl krb5kdc[23012](info): commencing operation
krb5kdc: starting...
W20260925 22:45:11.711391 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:45:11 dist-test-slave-csnl krb5kdc[23012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:45:11.838692 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36465
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:11.953496 23025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:11.953876 23025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:11.953958 23025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:11.958344 23025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:11.958494 23025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:11.958534 23025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:11.958582 23025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:11.958621 23025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:11.964390 23025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:11.965662 23025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:11.966777 23025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:45:11.976014 23025 server_base.cc:1034] running on GCE node
W20260925 22:45:11.975962 23031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:11.976033 23030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:11.975967 23033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:11.976897 23025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:11.978196 23025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:11.979404 23025 hybrid_clock.cc:648] HybridClock initialized: now 1790376311979392 us; error 42 us; skew 500 ppm
Sep 25 22:45:11 dist-test-slave-csnl krb5kdc[23012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:11.986660 23025 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:45:11.987828 23025 webserver.cc:460] Webserver started at http://127.0.0.1:39379/ using document root <none> and password file <none>
I20260925 22:45:11.988479 23025 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:11.988602 23025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:11.988864 23025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:11.991374 23025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "6096de79882c456dbffc8473631292e6"
format_stamp: "Formatted at 2026-09-25 22:45:11 on dist-test-slave-csnl"
server_key: "ecda8a6b89b7cb1d57501bc9265ead47"
server_key_iv: "e7b0c13599ff5561655c2a0fabf118e8"
server_key_version: "encryptionkey@0"
I20260925 22:45:11.992013 23025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "6096de79882c456dbffc8473631292e6"
format_stamp: "Formatted at 2026-09-25 22:45:11 on dist-test-slave-csnl"
server_key: "ecda8a6b89b7cb1d57501bc9265ead47"
server_key_iv: "e7b0c13599ff5561655c2a0fabf118e8"
server_key_version: "encryptionkey@0"
I20260925 22:45:11.996170 23025 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:45:11.998901 23040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:12.000340 23025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:12.000509 23025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "6096de79882c456dbffc8473631292e6"
format_stamp: "Formatted at 2026-09-25 22:45:11 on dist-test-slave-csnl"
server_key: "ecda8a6b89b7cb1d57501bc9265ead47"
server_key_iv: "e7b0c13599ff5561655c2a0fabf118e8"
server_key_version: "encryptionkey@0"
I20260925 22:45:12.000653 23025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:12.044718 23025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:12.048224 23025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:12.048408 23025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:12.056367 23025 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36465
I20260925 22:45:12.056389 23092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36465 every 8 connection(s)
I20260925 22:45:12.057423 23025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:12.060381 23093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:12.066219 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 23025
I20260925 22:45:12.066359 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:45:12.066596 23093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6096de79882c456dbffc8473631292e6: Bootstrap starting.
I20260925 22:45:12.066658 2347 external_mini_cluster.cc:1428] Setting key c6f0a041a39de1377d7a31e30c74876d
I20260925 22:45:12.069347 23093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6096de79882c456dbffc8473631292e6: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:12.070148 23093 log.cc:824] T 00000000000000000000000000000000 P 6096de79882c456dbffc8473631292e6: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:12.072414 23093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6096de79882c456dbffc8473631292e6: No bootstrap required, opened a new log
I20260925 22:45:12.076622 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:12.068205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57491 (local address 127.0.0.1:36465)
0925 22:45:12.068685 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:12.068700 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:12.068736 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:45:12.069426 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:12.069445 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:12.069887 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:12.070254 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:12.070819 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:12.071983 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:12.073064 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:12.073564 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:12.073769 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:12.074701 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:12.074707 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:12.074723 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:12.075240 (+ 517us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:12.075301 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:12.075327 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:12.075446 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:12.075588 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 22:45:12.075815 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":134,"threads_started":1}
I20260925 22:45:12.076448 23093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6096de79882c456dbffc8473631292e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6096de79882c456dbffc8473631292e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36465 } }
I20260925 22:45:12.076761 23093 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6096de79882c456dbffc8473631292e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:12.076828 23093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6096de79882c456dbffc8473631292e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6096de79882c456dbffc8473631292e6, State: Initialized, Role: FOLLOWER
I20260925 22:45:12.077358 23093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6096de79882c456dbffc8473631292e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6096de79882c456dbffc8473631292e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36465 } }
I20260925 22:45:12.077531 23093 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6096de79882c456dbffc8473631292e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:12.077607 23093 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6096de79882c456dbffc8473631292e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:12.077706 23093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6096de79882c456dbffc8473631292e6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:12.078830 23093 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6096de79882c456dbffc8473631292e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6096de79882c456dbffc8473631292e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36465 } }
I20260925 22:45:12.079263 23093 leader_election.cc:302] T 00000000000000000000000000000000 P 6096de79882c456dbffc8473631292e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6096de79882c456dbffc8473631292e6; no voters:
I20260925 22:45:12.079645 23093 leader_election.cc:288] T 00000000000000000000000000000000 P 6096de79882c456dbffc8473631292e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:12.079752 23098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6096de79882c456dbffc8473631292e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:12.079941 2347 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:45:12.080014 23098 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6096de79882c456dbffc8473631292e6 [term 1 LEADER]: Becoming Leader. State: Replica: 6096de79882c456dbffc8473631292e6, State: Running, Role: LEADER
I20260925 22:45:12.080418 23098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6096de79882c456dbffc8473631292e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6096de79882c456dbffc8473631292e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36465 } }
I20260925 22:45:12.081055 23093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6096de79882c456dbffc8473631292e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:12.082492 23100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6096de79882c456dbffc8473631292e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6096de79882c456dbffc8473631292e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6096de79882c456dbffc8473631292e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36465 } } }
I20260925 22:45:12.082502 23099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6096de79882c456dbffc8473631292e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6096de79882c456dbffc8473631292e6. Latest consensus state: current_term: 1 leader_uuid: "6096de79882c456dbffc8473631292e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6096de79882c456dbffc8473631292e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36465 } } }
I20260925 22:45:12.082679 23100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6096de79882c456dbffc8473631292e6 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:12.082677 23099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6096de79882c456dbffc8473631292e6 [sys.catalog]: This master's current role is: LEADER
W20260925 22:45:12.086768 23117 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6096de79882c456dbffc8473631292e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:45:12.086882 23117 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:45:12.086982 23112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:12.087932 23112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:12.089330 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:12.084246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51903 (local address 127.0.0.1:36465)
0925 22:45:12.084360 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:12.084364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:12.084634 (+ 270us) server_negotiation.cc:407] Connection header received
0925 22:45:12.084824 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:12.084828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:12.084888 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:12.084987 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:12.085400 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:12.086161 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:12.087354 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:12.087684 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:12.087766 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:12.088845 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:12.088849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:12.088852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:12.088868 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:12.088899 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:12.088912 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:12.088957 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:12.089122 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 22:45:12.089190 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 22:45:12.095525 23112 catalog_manager.cc:1347] Generated new cluster ID: ea5cffd8b2cb4b89b84c7d2def39c020
I20260925 22:45:12.095626 23112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:12.111869 23112 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:12.112970 23112 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:12.129227 23112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6096de79882c456dbffc8473631292e6: Generated new TSK 0
I20260925 22:45:12.130046 23112 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:45:12.134872 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 23025
2026-09-25T22:45:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1465 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:12 dist-test-slave-csnl krb5kdc[23124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:12 dist-test-slave-csnl krb5kdc[23124](info): set up 2 sockets
Sep 25 22:45:12 dist-test-slave-csnl krb5kdc[23124](info): commencing operation
krb5kdc: starting...
W20260925 22:45:13.179301 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:45:13 dist-test-slave-csnl krb5kdc[23124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:45:13.308336 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37683
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:13.427920 23137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:13.428298 23137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:13.428390 23137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:13.432690 23137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:13.432828 23137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:13.432869 23137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:13.432915 23137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:13.432948 23137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:13.438707 23137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:13.439975 23137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:13.441085 23137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:13.450536 23142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:13.450695 23137 server_base.cc:1034] running on GCE node
W20260925 22:45:13.450559 23145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:13.450546 23143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:13.451671 23137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:13.452894 23137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:13.454069 23137 hybrid_clock.cc:648] HybridClock initialized: now 1790376313454054 us; error 37 us; skew 500 ppm
Sep 25 22:45:13 dist-test-slave-csnl krb5kdc[23124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:13.457051 23137 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:45:13.458168 23137 webserver.cc:460] Webserver started at http://127.0.0.1:36083/ using document root <none> and password file <none>
I20260925 22:45:13.458729 23137 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:13.458793 23137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:13.459035 23137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:13.461265 23137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "792f49e9477a4af08e341c0fefd1f896"
format_stamp: "Formatted at 2026-09-25 22:45:13 on dist-test-slave-csnl"
server_key: "071f7573f5c5984874af0d89a6e17788"
server_key_iv: "863c5da2c196732f84321d7d33573821"
server_key_version: "encryptionkey@0"
I20260925 22:45:13.461829 23137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "792f49e9477a4af08e341c0fefd1f896"
format_stamp: "Formatted at 2026-09-25 22:45:13 on dist-test-slave-csnl"
server_key: "071f7573f5c5984874af0d89a6e17788"
server_key_iv: "863c5da2c196732f84321d7d33573821"
server_key_version: "encryptionkey@0"
I20260925 22:45:13.465967 23137 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 22:45:13.468822 23152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:13.470204 23137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:45:13.470335 23137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "792f49e9477a4af08e341c0fefd1f896"
format_stamp: "Formatted at 2026-09-25 22:45:13 on dist-test-slave-csnl"
server_key: "071f7573f5c5984874af0d89a6e17788"
server_key_iv: "863c5da2c196732f84321d7d33573821"
server_key_version: "encryptionkey@0"
I20260925 22:45:13.470448 23137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:13.475498 23137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:13.478714 23137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:13.478884 23137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:13.486717 23137 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37683
I20260925 22:45:13.486827 23204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37683 every 8 connection(s)
I20260925 22:45:13.487829 23137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:13.491286 23205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:13.495405 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 23137
I20260925 22:45:13.495579 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:45:13.495899 2347 external_mini_cluster.cc:1428] Setting key 2d355f59dfefb2625e8527a38ccb5da2
I20260925 22:45:13.498536 23205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 792f49e9477a4af08e341c0fefd1f896: Bootstrap starting.
I20260925 22:45:13.501327 23205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 792f49e9477a4af08e341c0fefd1f896: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:13.502184 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:13.497566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46103 (local address 127.0.0.1:37683)
0925 22:45:13.498148 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:13.498164 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:13.498206 (+ 42us) server_negotiation.cc:407] Connection header received
0925 22:45:13.499095 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:13.499118 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:13.499447 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:13.499832 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:13.500717 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:13.500722 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:13.500735 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:13.500746 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:13.500803 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:13.500823 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:13.501053 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:13.501183 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 22:45:13.501358 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":164,"threads_started":1}
I20260925 22:45:13.502246 23205 log.cc:824] T 00000000000000000000000000000000 P 792f49e9477a4af08e341c0fefd1f896: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:13.504864 23205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 792f49e9477a4af08e341c0fefd1f896: No bootstrap required, opened a new log
I20260925 22:45:13.505748 2347 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:45:13.509924 23205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 792f49e9477a4af08e341c0fefd1f896 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792f49e9477a4af08e341c0fefd1f896" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37683 } }
I20260925 22:45:13.510309 23205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 792f49e9477a4af08e341c0fefd1f896 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:13.510388 23205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 792f49e9477a4af08e341c0fefd1f896 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 792f49e9477a4af08e341c0fefd1f896, State: Initialized, Role: FOLLOWER
I20260925 22:45:13.510636 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:13.508214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53141 (local address 127.0.0.1:37683)
0925 22:45:13.508334 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:13.508338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:13.508485 (+ 147us) server_negotiation.cc:407] Connection header received
0925 22:45:13.508649 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:13.508652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:13.508688 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:13.508760 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:13.510103 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:13.510106 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:13.510108 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:13.510121 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:13.510154 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:13.510166 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:13.510257 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:13.510377 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 22:45:13.510425 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 22:45:13.511087 23205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 792f49e9477a4af08e341c0fefd1f896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792f49e9477a4af08e341c0fefd1f896" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37683 } }
I20260925 22:45:13.511287 23205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 792f49e9477a4af08e341c0fefd1f896 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:13.511350 23205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 792f49e9477a4af08e341c0fefd1f896 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:13.511459 23205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 792f49e9477a4af08e341c0fefd1f896 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:13.513253 23205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 792f49e9477a4af08e341c0fefd1f896 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792f49e9477a4af08e341c0fefd1f896" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37683 } }
I20260925 22:45:13.513645 23205 leader_election.cc:302] T 00000000000000000000000000000000 P 792f49e9477a4af08e341c0fefd1f896 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 792f49e9477a4af08e341c0fefd1f896; no voters:
I20260925 22:45:13.513968 23205 leader_election.cc:288] T 00000000000000000000000000000000 P 792f49e9477a4af08e341c0fefd1f896 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:13.514179 23215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 792f49e9477a4af08e341c0fefd1f896 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:13.514451 23215 raft_consensus.cc:686] T 00000000000000000000000000000000 P 792f49e9477a4af08e341c0fefd1f896 [term 1 LEADER]: Becoming Leader. State: Replica: 792f49e9477a4af08e341c0fefd1f896, State: Running, Role: LEADER
I20260925 22:45:13.514812 23215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 792f49e9477a4af08e341c0fefd1f896 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792f49e9477a4af08e341c0fefd1f896" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37683 } }
I20260925 22:45:13.515502 23205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 792f49e9477a4af08e341c0fefd1f896 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:13.516922 23216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 792f49e9477a4af08e341c0fefd1f896 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "792f49e9477a4af08e341c0fefd1f896" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792f49e9477a4af08e341c0fefd1f896" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37683 } } }
I20260925 22:45:13.517076 23216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 792f49e9477a4af08e341c0fefd1f896 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:13.517517 23224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:13.517483 23217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 792f49e9477a4af08e341c0fefd1f896 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 792f49e9477a4af08e341c0fefd1f896. Latest consensus state: current_term: 1 leader_uuid: "792f49e9477a4af08e341c0fefd1f896" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792f49e9477a4af08e341c0fefd1f896" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37683 } } }
I20260925 22:45:13.517629 23217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 792f49e9477a4af08e341c0fefd1f896 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:13.520781 23224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:13.527922 23224 catalog_manager.cc:1347] Generated new cluster ID: 8baaf381e7044058b0ad3cfb057d2b0f
I20260925 22:45:13.528028 23224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:13.541198 23224 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:13.541612 23224 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:13.558677 23224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 792f49e9477a4af08e341c0fefd1f896: Generated new TSK 0
I20260925 22:45:13.559711 23224 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:45:13.573102 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 23137
2026-09-25T22:45:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:13 dist-test-slave-csnl krb5kdc[23236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:13 dist-test-slave-csnl krb5kdc[23236](info): set up 2 sockets
Sep 25 22:45:13 dist-test-slave-csnl krb5kdc[23236](info): commencing operation
krb5kdc: starting...
W20260925 22:45:14.615345 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:45:14 dist-test-slave-csnl krb5kdc[23236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:45:14.746883 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34131
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:14.872828 23249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:14.873157 23249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:14.873203 23249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:14.877545 23249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:14.877638 23249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:14.877658 23249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:14.877676 23249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:14.877692 23249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:14.883526 23249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:14.884758 23249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:14.885891 23249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:14.895365 23254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:14.895344 23257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:14.895344 23255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:14.895346 23249 server_base.cc:1034] running on GCE node
I20260925 22:45:14.896248 23249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:14.897581 23249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:14.898777 23249 hybrid_clock.cc:648] HybridClock initialized: now 1790376314898740 us; error 55 us; skew 500 ppm
Sep 25 22:45:14 dist-test-slave-csnl krb5kdc[23236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:14.901876 23249 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:45:14.903036 23249 webserver.cc:460] Webserver started at http://127.0.0.1:37247/ using document root <none> and password file <none>
I20260925 22:45:14.903673 23249 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:14.903746 23249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:14.903978 23249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:14.906232 23249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "1b12cac23d754f93b508115ab826ef75"
format_stamp: "Formatted at 2026-09-25 22:45:14 on dist-test-slave-csnl"
server_key: "060fde2f6fadbe95577f260531996380"
server_key_iv: "1892402a853b1daa7056ef8e1a204029"
server_key_version: "encryptionkey@0"
I20260925 22:45:14.906893 23249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "1b12cac23d754f93b508115ab826ef75"
format_stamp: "Formatted at 2026-09-25 22:45:14 on dist-test-slave-csnl"
server_key: "060fde2f6fadbe95577f260531996380"
server_key_iv: "1892402a853b1daa7056ef8e1a204029"
server_key_version: "encryptionkey@0"
I20260925 22:45:14.911202 23249 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:45:14.914074 23264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:14.915594 23249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:14.915751 23249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "1b12cac23d754f93b508115ab826ef75"
format_stamp: "Formatted at 2026-09-25 22:45:14 on dist-test-slave-csnl"
server_key: "060fde2f6fadbe95577f260531996380"
server_key_iv: "1892402a853b1daa7056ef8e1a204029"
server_key_version: "encryptionkey@0"
I20260925 22:45:14.915889 23249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:14.949898 23249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:14.953471 23249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:14.953704 23249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:14.961410 23249 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34131
I20260925 22:45:14.961426 23316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34131 every 8 connection(s)
I20260925 22:45:14.962467 23249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:14.965042 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 23249
I20260925 22:45:14.965209 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:45:14.965529 2347 external_mini_cluster.cc:1428] Setting key 2c25f405458794bf7d550c2f1bb349aa
I20260925 22:45:14.965551 23317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:14.973167 23317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b12cac23d754f93b508115ab826ef75: Bootstrap starting.
Sep 25 22:45:14 dist-test-slave-csnl krb5kdc[23236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 22:45:14.976219 23317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b12cac23d754f93b508115ab826ef75: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:14.977039 23317 log.cc:824] T 00000000000000000000000000000000 P 1b12cac23d754f93b508115ab826ef75: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:14.979278 23317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b12cac23d754f93b508115ab826ef75: No bootstrap required, opened a new log
I20260925 22:45:14.981480 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:14.967137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:51985 (local address 127.0.0.1:34131)
0925 22:45:14.967587 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:14.967606 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:14.967652 (+ 46us) server_negotiation.cc:407] Connection header received
0925 22:45:14.968512 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:14.968533 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:14.968913 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:14.969215 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:14.970235 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:14.971370 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:14.972566 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:14.973067 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:14.973193 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:14.976185 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:14.976191 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:14.976207 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:14.976267 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:14.978162 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:14.978764 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:14.978769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:14.978774 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:14.978889 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:14.979299 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:14.979309 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:14.979311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:14.979758 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:14.979978 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:14.980312 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 22:45:14.980545 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":163,"threads_started":1}
I20260925 22:45:14.983767 23317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b12cac23d754f93b508115ab826ef75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b12cac23d754f93b508115ab826ef75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34131 } }
I20260925 22:45:14.984114 23317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b12cac23d754f93b508115ab826ef75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:14.984181 23317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b12cac23d754f93b508115ab826ef75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b12cac23d754f93b508115ab826ef75, State: Initialized, Role: FOLLOWER
I20260925 22:45:14.984748 23317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b12cac23d754f93b508115ab826ef75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b12cac23d754f93b508115ab826ef75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34131 } }
I20260925 22:45:14.984929 23317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b12cac23d754f93b508115ab826ef75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:14.984980 2347 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:45:14.985005 23317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b12cac23d754f93b508115ab826ef75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:14.985127 23317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b12cac23d754f93b508115ab826ef75 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:14.986205 23317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b12cac23d754f93b508115ab826ef75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b12cac23d754f93b508115ab826ef75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34131 } }
I20260925 22:45:14.986625 23317 leader_election.cc:302] T 00000000000000000000000000000000 P 1b12cac23d754f93b508115ab826ef75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b12cac23d754f93b508115ab826ef75; no voters:
I20260925 22:45:14.987026 23317 leader_election.cc:288] T 00000000000000000000000000000000 P 1b12cac23d754f93b508115ab826ef75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:14.987140 23326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b12cac23d754f93b508115ab826ef75 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:14.987376 23326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b12cac23d754f93b508115ab826ef75 [term 1 LEADER]: Becoming Leader. State: Replica: 1b12cac23d754f93b508115ab826ef75, State: Running, Role: LEADER
I20260925 22:45:14.987771 23326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b12cac23d754f93b508115ab826ef75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b12cac23d754f93b508115ab826ef75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34131 } }
I20260925 22:45:14.988425 23317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b12cac23d754f93b508115ab826ef75 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:14.989042 23326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b12cac23d754f93b508115ab826ef75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b12cac23d754f93b508115ab826ef75. Latest consensus state: current_term: 1 leader_uuid: "1b12cac23d754f93b508115ab826ef75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b12cac23d754f93b508115ab826ef75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34131 } } }
I20260925 22:45:14.989231 23326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b12cac23d754f93b508115ab826ef75 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:14.989562 23328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b12cac23d754f93b508115ab826ef75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b12cac23d754f93b508115ab826ef75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b12cac23d754f93b508115ab826ef75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34131 } } }
I20260925 22:45:14.989677 23335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:14.989745 23328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b12cac23d754f93b508115ab826ef75 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:14.993462 23335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:15.000478 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:14.987243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:34977 (local address 127.0.0.1:34131)
0925 22:45:14.987379 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:14.987386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:14.990983 (+ 3597us) server_negotiation.cc:407] Connection header received
0925 22:45:14.991206 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:14.991209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:14.991269 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:14.991377 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:14.992238 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:14.992775 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:14.993998 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:14.994357 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:14.994431 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:14.996419 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:14.996424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:14.996427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:14.996459 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:14.998407 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:14.999040 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:14.999044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:14.999045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:14.999143 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:14.999522 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:14.999525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:14.999527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:14.999720 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:14.999890 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:15.000170 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 22:45:15.000281 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 22:45:15.000914 23335 catalog_manager.cc:1347] Generated new cluster ID: 6a84458b625f474a87227885283ca9aa
I20260925 22:45:15.000980 23335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:15.009828 23335 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:15.011330 23335 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:15.021324 23335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b12cac23d754f93b508115ab826ef75: Generated new TSK 0
I20260925 22:45:15.022655 23335 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:45:15.034255 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 23249
W20260925 22:45:15.041278 23234 connection.cc:535] client connection to 127.0.0.1:34131 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34131): Connection reset by peer (error 104)
2026-09-25T22:45:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1462 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:15 dist-test-slave-csnl krb5kdc[23348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:15 dist-test-slave-csnl krb5kdc[23348](info): set up 2 sockets
Sep 25 22:45:15 dist-test-slave-csnl krb5kdc[23348](info): commencing operation
krb5kdc: starting...
W20260925 22:45:16.079326 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:45:16 dist-test-slave-csnl krb5kdc[23348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:45:16.213728 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45369
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:16.332247 23361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:16.332631 23361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:16.332719 23361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:16.337567 23361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:16.337725 23361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:16.337769 23361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:16.337822 23361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:16.337901 23361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:16.344225 23361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:16.345487 23361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:16.346632 23361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:16.356104 23366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:16.356101 23367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:16.356117 23369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:16.356258 23361 server_base.cc:1034] running on GCE node
I20260925 22:45:16.357225 23361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:16.358652 23361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:16.359864 23361 hybrid_clock.cc:648] HybridClock initialized: now 1790376316359850 us; error 87 us; skew 500 ppm
Sep 25 22:45:16 dist-test-slave-csnl krb5kdc[23348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:16.363037 23361 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:45:16.364099 23361 webserver.cc:460] Webserver started at http://127.0.0.1:37229/ using document root <none> and password file <none>
I20260925 22:45:16.364741 23361 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:16.364851 23361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:16.365113 23361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:16.367475 23361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "ba5613425f5e43e4838ef7a095e53f9e"
format_stamp: "Formatted at 2026-09-25 22:45:16 on dist-test-slave-csnl"
server_key: "a7763ad7f61581509653a9dc4150f8a7"
server_key_iv: "6e768ac3f7b949c871a0d707cf6b4355"
server_key_version: "encryptionkey@0"
I20260925 22:45:16.368146 23361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "ba5613425f5e43e4838ef7a095e53f9e"
format_stamp: "Formatted at 2026-09-25 22:45:16 on dist-test-slave-csnl"
server_key: "a7763ad7f61581509653a9dc4150f8a7"
server_key_iv: "6e768ac3f7b949c871a0d707cf6b4355"
server_key_version: "encryptionkey@0"
I20260925 22:45:16.372686 23361 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:45:16.375695 23376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:16.377282 23361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:16.377458 23361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "ba5613425f5e43e4838ef7a095e53f9e"
format_stamp: "Formatted at 2026-09-25 22:45:16 on dist-test-slave-csnl"
server_key: "a7763ad7f61581509653a9dc4150f8a7"
server_key_iv: "6e768ac3f7b949c871a0d707cf6b4355"
server_key_version: "encryptionkey@0"
I20260925 22:45:16.377673 23361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:16.389985 23361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:16.393445 23361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:16.393633 23361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:16.401650 23361 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45369
I20260925 22:45:16.401664 23428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45369 every 8 connection(s)
I20260925 22:45:16.402751 23361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:16.405846 23429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:16.411303 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 23361
I20260925 22:45:16.411471 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:45:16.411753 2347 external_mini_cluster.cc:1428] Setting key 8d5c10fddc3fab7abc7983f66b7ad28d
I20260925 22:45:16.412690 23429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba5613425f5e43e4838ef7a095e53f9e: Bootstrap starting.
I20260925 22:45:16.415350 23429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba5613425f5e43e4838ef7a095e53f9e: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:16.416195 23429 log.cc:824] T 00000000000000000000000000000000 P ba5613425f5e43e4838ef7a095e53f9e: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:16.418440 23429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba5613425f5e43e4838ef7a095e53f9e: No bootstrap required, opened a new log
I20260925 22:45:16.421850 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:16.413396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:35115 (local address 127.0.0.1:45369)
0925 22:45:16.413881 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:16.413899 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:16.413939 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:45:16.414679 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:16.414699 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:16.415189 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:16.415612 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:16.416082 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:16.417180 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:16.418191 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:16.418763 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:16.418948 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:16.419963 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:16.419969 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:16.419984 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:16.420411 (+ 427us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:16.420472 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:16.420497 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:16.420667 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:16.420866 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 22:45:16.421033 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":159,"threads_started":1}
I20260925 22:45:16.423512 23429 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba5613425f5e43e4838ef7a095e53f9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5613425f5e43e4838ef7a095e53f9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45369 } }
I20260925 22:45:16.423808 23429 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba5613425f5e43e4838ef7a095e53f9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:16.423897 23429 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba5613425f5e43e4838ef7a095e53f9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba5613425f5e43e4838ef7a095e53f9e, State: Initialized, Role: FOLLOWER
I20260925 22:45:16.424417 23429 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba5613425f5e43e4838ef7a095e53f9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5613425f5e43e4838ef7a095e53f9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45369 } }
I20260925 22:45:16.424590 23429 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba5613425f5e43e4838ef7a095e53f9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:16.424686 23429 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba5613425f5e43e4838ef7a095e53f9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:16.424814 23429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba5613425f5e43e4838ef7a095e53f9e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:16.425690 2347 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:45:16.426112 23429 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba5613425f5e43e4838ef7a095e53f9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5613425f5e43e4838ef7a095e53f9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45369 } }
I20260925 22:45:16.426502 23429 leader_election.cc:302] T 00000000000000000000000000000000 P ba5613425f5e43e4838ef7a095e53f9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba5613425f5e43e4838ef7a095e53f9e; no voters:
I20260925 22:45:16.426934 23429 leader_election.cc:288] T 00000000000000000000000000000000 P ba5613425f5e43e4838ef7a095e53f9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:16.427127 23435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba5613425f5e43e4838ef7a095e53f9e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:16.427359 23435 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba5613425f5e43e4838ef7a095e53f9e [term 1 LEADER]: Becoming Leader. State: Replica: ba5613425f5e43e4838ef7a095e53f9e, State: Running, Role: LEADER
I20260925 22:45:16.427752 23435 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba5613425f5e43e4838ef7a095e53f9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5613425f5e43e4838ef7a095e53f9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45369 } }
I20260925 22:45:16.428519 23429 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba5613425f5e43e4838ef7a095e53f9e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:16.429823 23439 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba5613425f5e43e4838ef7a095e53f9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba5613425f5e43e4838ef7a095e53f9e. Latest consensus state: current_term: 1 leader_uuid: "ba5613425f5e43e4838ef7a095e53f9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5613425f5e43e4838ef7a095e53f9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45369 } } }
I20260925 22:45:16.429970 23439 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba5613425f5e43e4838ef7a095e53f9e [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:16.430289 23440 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba5613425f5e43e4838ef7a095e53f9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba5613425f5e43e4838ef7a095e53f9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5613425f5e43e4838ef7a095e53f9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45369 } } }
I20260925 22:45:16.430387 23440 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba5613425f5e43e4838ef7a095e53f9e [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:16.430843 23448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:16.434165 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:16.428161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:32905 (local address 127.0.0.1:45369)
0925 22:45:16.428316 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:16.428322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:16.428513 (+ 191us) server_negotiation.cc:407] Connection header received
0925 22:45:16.428685 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:16.428688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:16.428741 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:16.428791 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:16.430112 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:16.430890 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:16.432245 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:16.432500 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:16.432622 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:16.433623 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:16.433627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:16.433630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:16.433652 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:16.433680 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:16.433692 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:16.433793 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:16.433965 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 22:45:16.434005 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 22:45:16.434600 23448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:16.440855 23448 catalog_manager.cc:1347] Generated new cluster ID: 115a73b8e5dd484db6e525bb9d9e19f2
I20260925 22:45:16.440953 23448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:16.467300 23448 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:16.468653 23448 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:16.474851 23448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba5613425f5e43e4838ef7a095e53f9e: Generated new TSK 0
I20260925 22:45:16.475744 23448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:45:16.481395 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 23361
W20260925 22:45:16.487017 23346 connection.cc:535] client connection to 127.0.0.1:45369 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45369): Connection reset by peer (error 104)
2026-09-25T22:45:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:16 dist-test-slave-csnl krb5kdc[23460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:16 dist-test-slave-csnl krb5kdc[23460](info): set up 2 sockets
Sep 25 22:45:16 dist-test-slave-csnl krb5kdc[23460](info): commencing operation
krb5kdc: starting...
W20260925 22:45:17.535283 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:45:17 dist-test-slave-csnl krb5kdc[23460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:45:17.662410 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39017
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:17.775764 23473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:17.776083 23473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:17.776129 23473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:17.780346 23473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:17.780427 23473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:17.780448 23473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:17.780465 23473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:17.780481 23473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:17.786195 23473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:17.787358 23473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:17.788410 23473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:17.797744 23481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:17.797876 23473 server_base.cc:1034] running on GCE node
W20260925 22:45:17.797744 23478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:17.797744 23479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:17.798677 23473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:17.799999 23473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:17.801219 23473 hybrid_clock.cc:648] HybridClock initialized: now 1790376317801182 us; error 53 us; skew 500 ppm
Sep 25 22:45:17 dist-test-slave-csnl krb5kdc[23460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:17.810547 23473 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:45:17.811708 23473 webserver.cc:460] Webserver started at http://127.0.0.1:37013/ using document root <none> and password file <none>
I20260925 22:45:17.812350 23473 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:17.812457 23473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:17.812716 23473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:17.815001 23473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "2122a27bde9f447da460858261ff5e42"
format_stamp: "Formatted at 2026-09-25 22:45:17 on dist-test-slave-csnl"
server_key: "7f245117cc6593d67138efd4149da859"
server_key_iv: "e38aceb3a781dc17ce57c5347a769a1b"
server_key_version: "encryptionkey@0"
I20260925 22:45:17.815667 23473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "2122a27bde9f447da460858261ff5e42"
format_stamp: "Formatted at 2026-09-25 22:45:17 on dist-test-slave-csnl"
server_key: "7f245117cc6593d67138efd4149da859"
server_key_iv: "e38aceb3a781dc17ce57c5347a769a1b"
server_key_version: "encryptionkey@0"
I20260925 22:45:17.819929 23473 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:45:17.822680 23488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:17.824141 23473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:17.824316 23473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "2122a27bde9f447da460858261ff5e42"
format_stamp: "Formatted at 2026-09-25 22:45:17 on dist-test-slave-csnl"
server_key: "7f245117cc6593d67138efd4149da859"
server_key_iv: "e38aceb3a781dc17ce57c5347a769a1b"
server_key_version: "encryptionkey@0"
I20260925 22:45:17.824479 23473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:17.829439 23473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:17.833201 23473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:17.833427 23473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:17.840788 23473 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39017
I20260925 22:45:17.840889 23540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39017 every 8 connection(s)
I20260925 22:45:17.841893 23473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:17.844990 23541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:17.849306 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 23473
I20260925 22:45:17.849481 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:45:17.849776 2347 external_mini_cluster.cc:1428] Setting key 550e7b3de64fb9fc5b12c5fe3eb78273
I20260925 22:45:17.851485 23541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2122a27bde9f447da460858261ff5e42: Bootstrap starting.
I20260925 22:45:17.853922 23541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2122a27bde9f447da460858261ff5e42: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:17.854924 23541 log.cc:824] T 00000000000000000000000000000000 P 2122a27bde9f447da460858261ff5e42: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:17.855706 23544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:17.851357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:41235 (local address 127.0.0.1:39017)
0925 22:45:17.851893 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:17.851909 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:17.851949 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:45:17.852655 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:17.852672 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:17.852973 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:17.853344 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:17.854201 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:17.854210 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:17.854229 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:17.854246 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:17.854320 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:17.854349 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:17.854473 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:17.854624 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 22:45:17.854789 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":163,"threads_started":1}
I20260925 22:45:17.857137 23541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2122a27bde9f447da460858261ff5e42: No bootstrap required, opened a new log
I20260925 22:45:17.858702 2347 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:45:17.861719 23541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2122a27bde9f447da460858261ff5e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2122a27bde9f447da460858261ff5e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39017 } }
I20260925 22:45:17.862042 23541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2122a27bde9f447da460858261ff5e42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:17.862111 23541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2122a27bde9f447da460858261ff5e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2122a27bde9f447da460858261ff5e42, State: Initialized, Role: FOLLOWER
I20260925 22:45:17.862586 23541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2122a27bde9f447da460858261ff5e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2122a27bde9f447da460858261ff5e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39017 } }
I20260925 22:45:17.862715 23544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:17.860881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:56879 (local address 127.0.0.1:39017)
0925 22:45:17.861003 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:17.861007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:17.861224 (+ 217us) server_negotiation.cc:407] Connection header received
0925 22:45:17.861345 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:17.861347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:17.861378 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:17.861445 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:17.862310 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:17.862314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:17.862316 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:17.862329 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:17.862356 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:17.862376 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:17.862469 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:17.862541 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 22:45:17.862583 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:45:17.862787 23541 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2122a27bde9f447da460858261ff5e42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:17.862854 23541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2122a27bde9f447da460858261ff5e42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:17.862975 23541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2122a27bde9f447da460858261ff5e42 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:17.864066 23541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2122a27bde9f447da460858261ff5e42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2122a27bde9f447da460858261ff5e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39017 } }
I20260925 22:45:17.864420 23541 leader_election.cc:302] T 00000000000000000000000000000000 P 2122a27bde9f447da460858261ff5e42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2122a27bde9f447da460858261ff5e42; no voters:
I20260925 22:45:17.864740 23541 leader_election.cc:288] T 00000000000000000000000000000000 P 2122a27bde9f447da460858261ff5e42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:17.864934 23551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2122a27bde9f447da460858261ff5e42 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:17.865172 23551 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2122a27bde9f447da460858261ff5e42 [term 1 LEADER]: Becoming Leader. State: Replica: 2122a27bde9f447da460858261ff5e42, State: Running, Role: LEADER
I20260925 22:45:17.865487 23551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2122a27bde9f447da460858261ff5e42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2122a27bde9f447da460858261ff5e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39017 } }
I20260925 22:45:17.866046 23541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2122a27bde9f447da460858261ff5e42 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:17.867197 23552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2122a27bde9f447da460858261ff5e42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2122a27bde9f447da460858261ff5e42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2122a27bde9f447da460858261ff5e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39017 } } }
I20260925 22:45:17.867336 23552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2122a27bde9f447da460858261ff5e42 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:17.867697 23553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2122a27bde9f447da460858261ff5e42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2122a27bde9f447da460858261ff5e42. Latest consensus state: current_term: 1 leader_uuid: "2122a27bde9f447da460858261ff5e42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2122a27bde9f447da460858261ff5e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39017 } } }
I20260925 22:45:17.867764 23560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:17.867789 23553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2122a27bde9f447da460858261ff5e42 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:17.870776 23560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:17.877377 23560 catalog_manager.cc:1347] Generated new cluster ID: d7b0d5ecefa840eb832287904f3be815
I20260925 22:45:17.877475 23560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:17.894167 23560 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:17.894649 23560 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:17.902879 23560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2122a27bde9f447da460858261ff5e42: Generated new TSK 0
I20260925 22:45:17.903769 23560 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:45:17.909513 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 23473
2026-09-25T22:45:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1428 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:17 dist-test-slave-csnl krb5kdc[23572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:17 dist-test-slave-csnl krb5kdc[23572](info): set up 2 sockets
Sep 25 22:45:17 dist-test-slave-csnl krb5kdc[23572](info): commencing operation
krb5kdc: starting...
W20260925 22:45:18.947237 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:45:19 dist-test-slave-csnl krb5kdc[23572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:45:19.079694 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35709
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:19.196830 23585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:19.197132 23585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:19.197177 23585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:19.201462 23585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:19.201551 23585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:19.201571 23585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:19.201589 23585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:19.201606 23585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:19.207420 23585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35709
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:19.208523 23585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:19.209549 23585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:19.218386 23591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:19.218386 23593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:19.218420 23590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:19.218633 23585 server_base.cc:1034] running on GCE node
I20260925 22:45:19.219462 23585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:19.220649 23585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:19.221817 23585 hybrid_clock.cc:648] HybridClock initialized: now 1790376319221808 us; error 31 us; skew 500 ppm
Sep 25 22:45:19 dist-test-slave-csnl krb5kdc[23572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:19.224712 23585 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:45:19.225729 23585 webserver.cc:460] Webserver started at http://127.0.0.1:34889/ using document root <none> and password file <none>
I20260925 22:45:19.226327 23585 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:19.226418 23585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:19.226652 23585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:19.228868 23585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "41b47488f35243b7a436ed1a56b5c061"
format_stamp: "Formatted at 2026-09-25 22:45:19 on dist-test-slave-csnl"
server_key: "1789866ec8977cb6c7ac7bfa5bb7643d"
server_key_iv: "4b5fdf67a2ac699b52824caba66ea90d"
server_key_version: "encryptionkey@0"
I20260925 22:45:19.229454 23585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "41b47488f35243b7a436ed1a56b5c061"
format_stamp: "Formatted at 2026-09-25 22:45:19 on dist-test-slave-csnl"
server_key: "1789866ec8977cb6c7ac7bfa5bb7643d"
server_key_iv: "4b5fdf67a2ac699b52824caba66ea90d"
server_key_version: "encryptionkey@0"
I20260925 22:45:19.233327 23585 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:45:19.235912 23600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:19.237136 23585 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:45:19.237280 23585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "41b47488f35243b7a436ed1a56b5c061"
format_stamp: "Formatted at 2026-09-25 22:45:19 on dist-test-slave-csnl"
server_key: "1789866ec8977cb6c7ac7bfa5bb7643d"
server_key_iv: "4b5fdf67a2ac699b52824caba66ea90d"
server_key_version: "encryptionkey@0"
I20260925 22:45:19.237403 23585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:19.260958 23585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:19.264802 23585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:19.265058 23585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:19.272902 23585 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35709
I20260925 22:45:19.272917 23652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35709 every 8 connection(s)
I20260925 22:45:19.273952 23585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:19.276768 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 23585
I20260925 22:45:19.276952 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:45:19.277101 23653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:19.277287 2347 external_mini_cluster.cc:1428] Setting key 3da3ac44e2bd569ced8651d0719d4e17
I20260925 22:45:19.284737 23653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41b47488f35243b7a436ed1a56b5c061: Bootstrap starting.
Sep 25 22:45:19 dist-test-slave-csnl krb5kdc[23572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 22:45:19.287834 23653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41b47488f35243b7a436ed1a56b5c061: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:19.288909 23653 log.cc:824] T 00000000000000000000000000000000 P 41b47488f35243b7a436ed1a56b5c061: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:19.291446 23653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41b47488f35243b7a436ed1a56b5c061: No bootstrap required, opened a new log
I20260925 22:45:19.293231 23656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:19.278651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:59503 (local address 127.0.0.1:35709)
0925 22:45:19.279116 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:19.279132 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:19.279167 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:45:19.280053 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:19.280072 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:19.280437 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:19.280725 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:19.281718 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:19.282747 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:19.283995 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:19.284542 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:19.284680 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:19.287718 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:19.287728 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:19.287751 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:19.287798 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:19.289724 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:19.290306 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:19.290312 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:19.290320 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:19.290473 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:19.290897 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:19.290900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:19.290902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:19.291458 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:19.291681 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:19.292002 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 22:45:19.292244 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":148,"threads_started":1}
I20260925 22:45:19.296190 23653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41b47488f35243b7a436ed1a56b5c061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b47488f35243b7a436ed1a56b5c061" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35709 } }
I20260925 22:45:19.296489 23653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41b47488f35243b7a436ed1a56b5c061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:19.296602 2347 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:45:19.296623 23653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41b47488f35243b7a436ed1a56b5c061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41b47488f35243b7a436ed1a56b5c061, State: Initialized, Role: FOLLOWER
I20260925 22:45:19.297091 23653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41b47488f35243b7a436ed1a56b5c061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b47488f35243b7a436ed1a56b5c061" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35709 } }
I20260925 22:45:19.297220 23653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41b47488f35243b7a436ed1a56b5c061 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:19.297305 23653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41b47488f35243b7a436ed1a56b5c061 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:19.297425 23653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41b47488f35243b7a436ed1a56b5c061 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:19.298619 23653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41b47488f35243b7a436ed1a56b5c061 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b47488f35243b7a436ed1a56b5c061" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35709 } }
I20260925 22:45:19.299086 23653 leader_election.cc:302] T 00000000000000000000000000000000 P 41b47488f35243b7a436ed1a56b5c061 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41b47488f35243b7a436ed1a56b5c061; no voters:
I20260925 22:45:19.299454 23653 leader_election.cc:288] T 00000000000000000000000000000000 P 41b47488f35243b7a436ed1a56b5c061 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:19.299533 23663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41b47488f35243b7a436ed1a56b5c061 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:19.299746 23663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41b47488f35243b7a436ed1a56b5c061 [term 1 LEADER]: Becoming Leader. State: Replica: 41b47488f35243b7a436ed1a56b5c061, State: Running, Role: LEADER
I20260925 22:45:19.300117 23663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41b47488f35243b7a436ed1a56b5c061 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b47488f35243b7a436ed1a56b5c061" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35709 } }
I20260925 22:45:19.300864 23653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41b47488f35243b7a436ed1a56b5c061 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:19.301867 23665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41b47488f35243b7a436ed1a56b5c061 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41b47488f35243b7a436ed1a56b5c061. Latest consensus state: current_term: 1 leader_uuid: "41b47488f35243b7a436ed1a56b5c061" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b47488f35243b7a436ed1a56b5c061" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35709 } } }
I20260925 22:45:19.302002 23665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41b47488f35243b7a436ed1a56b5c061 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:19.302323 23664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41b47488f35243b7a436ed1a56b5c061 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41b47488f35243b7a436ed1a56b5c061" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b47488f35243b7a436ed1a56b5c061" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35709 } } }
I20260925 22:45:19.302417 23664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41b47488f35243b7a436ed1a56b5c061 [sys.catalog]: This master's current role is: LEADER
W20260925 22:45:19.306130 23679 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 41b47488f35243b7a436ed1a56b5c061: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:45:19.306223 23679 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:45:19.306291 23672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:19.307273 23672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:19.308205 23656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:19.298787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:41465 (local address 127.0.0.1:35709)
0925 22:45:19.298914 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:19.298919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:19.299148 (+ 229us) server_negotiation.cc:407] Connection header received
0925 22:45:19.299290 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:19.299295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:19.299344 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:19.299428 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:19.300275 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:19.300818 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:19.301939 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:19.302264 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:19.302353 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:19.304255 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:19.304259 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:19.304262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:19.304289 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:19.305981 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:19.306615 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:19.306618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:19.306620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:19.306711 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:19.307179 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:19.307183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:19.307185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:19.307431 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:19.307590 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:19.307926 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 22:45:19.308043 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 22:45:19.313716 23672 catalog_manager.cc:1347] Generated new cluster ID: cf5596a6704445bb9f94d07f73e4d6cd
I20260925 22:45:19.313808 23672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:19.329876 23672 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:19.331344 23672 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:19.338060 23672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41b47488f35243b7a436ed1a56b5c061: Generated new TSK 0
I20260925 22:45:19.339063 23672 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:45:19.348994 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 23585
W20260925 22:45:19.358297 23570 connection.cc:535] client connection to 127.0.0.1:35709 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35709): Connection reset by peer (error 104)
2026-09-25T22:45:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1444 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:19 dist-test-slave-csnl krb5kdc[23684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:19 dist-test-slave-csnl krb5kdc[23684](info): set up 2 sockets
Sep 25 22:45:19 dist-test-slave-csnl krb5kdc[23684](info): commencing operation
krb5kdc: starting...
W20260925 22:45:20.419241 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:45:20 dist-test-slave-csnl krb5kdc[23684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:45:20.544427 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36819
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:20.663340 23697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:20.663694 23697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:20.663743 23697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:20.668124 23697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:20.668203 23697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:20.668224 23697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:20.668242 23697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:20.668287 23697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:20.674018 23697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36819
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:20.675225 23697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:20.676299 23697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:20.685149 23703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:20.685153 23702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:20.685153 23705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:20.685580 23697 server_base.cc:1034] running on GCE node
I20260925 22:45:20.686208 23697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:20.687453 23697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:20.688645 23697 hybrid_clock.cc:648] HybridClock initialized: now 1790376320688635 us; error 33 us; skew 500 ppm
Sep 25 22:45:20 dist-test-slave-csnl krb5kdc[23684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:20.691608 23697 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:45:20.692633 23697 webserver.cc:460] Webserver started at http://127.0.0.1:32923/ using document root <none> and password file <none>
I20260925 22:45:20.693225 23697 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:20.693315 23697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:20.693549 23697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:20.695815 23697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "dd13a563edca4cbb838ad02d9d61cd54"
format_stamp: "Formatted at 2026-09-25 22:45:20 on dist-test-slave-csnl"
server_key: "839f0790f3f4e761de95ac133930f578"
server_key_iv: "0e5c09965993bae0462b4a668bee54af"
server_key_version: "encryptionkey@0"
I20260925 22:45:20.696463 23697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "dd13a563edca4cbb838ad02d9d61cd54"
format_stamp: "Formatted at 2026-09-25 22:45:20 on dist-test-slave-csnl"
server_key: "839f0790f3f4e761de95ac133930f578"
server_key_iv: "0e5c09965993bae0462b4a668bee54af"
server_key_version: "encryptionkey@0"
I20260925 22:45:20.700752 23697 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:45:20.703569 23712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:20.704960 23697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:20.705153 23697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "dd13a563edca4cbb838ad02d9d61cd54"
format_stamp: "Formatted at 2026-09-25 22:45:20 on dist-test-slave-csnl"
server_key: "839f0790f3f4e761de95ac133930f578"
server_key_iv: "0e5c09965993bae0462b4a668bee54af"
server_key_version: "encryptionkey@0"
I20260925 22:45:20.705298 23697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:20.720395 23697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:20.723966 23697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:20.724196 23697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:20.731967 23697 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36819
I20260925 22:45:20.731987 23764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36819 every 8 connection(s)
I20260925 22:45:20.733016 23697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:20.736024 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:20.741813 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 23697
I20260925 22:45:20.741971 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:45:20.742255 2347 external_mini_cluster.cc:1428] Setting key a9b52dbad9decd4bf4bf8639131adf52
I20260925 22:45:20.742625 23765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd13a563edca4cbb838ad02d9d61cd54: Bootstrap starting.
I20260925 22:45:20.745489 23765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd13a563edca4cbb838ad02d9d61cd54: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:20.746315 23765 log.cc:824] T 00000000000000000000000000000000 P dd13a563edca4cbb838ad02d9d61cd54: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:20.748593 23765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd13a563edca4cbb838ad02d9d61cd54: No bootstrap required, opened a new log
I20260925 22:45:20.752301 23768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:20.743835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:45145 (local address 127.0.0.1:36819)
0925 22:45:20.744382 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:20.744398 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:20.744438 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:45:20.745136 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:20.745154 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:20.745560 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:20.745977 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:20.746430 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:20.747749 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:20.748695 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:20.749235 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:20.749478 (+ 243us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:20.750419 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:20.750425 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:20.750438 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:20.750879 (+ 441us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:20.750939 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:20.751031 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:20.751167 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:20.751355 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 22:45:20.751503 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":163,"threads_started":1}
I20260925 22:45:20.752838 23765 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd13a563edca4cbb838ad02d9d61cd54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd13a563edca4cbb838ad02d9d61cd54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36819 } }
I20260925 22:45:20.753134 23765 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd13a563edca4cbb838ad02d9d61cd54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:20.753201 23765 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd13a563edca4cbb838ad02d9d61cd54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd13a563edca4cbb838ad02d9d61cd54, State: Initialized, Role: FOLLOWER
I20260925 22:45:20.753675 23765 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd13a563edca4cbb838ad02d9d61cd54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd13a563edca4cbb838ad02d9d61cd54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36819 } }
I20260925 22:45:20.753813 23765 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd13a563edca4cbb838ad02d9d61cd54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:20.753902 23765 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd13a563edca4cbb838ad02d9d61cd54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:20.754046 23765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd13a563edca4cbb838ad02d9d61cd54 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:20.755362 23765 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd13a563edca4cbb838ad02d9d61cd54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd13a563edca4cbb838ad02d9d61cd54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36819 } }
I20260925 22:45:20.755759 2347 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:45:20.755748 23765 leader_election.cc:302] T 00000000000000000000000000000000 P dd13a563edca4cbb838ad02d9d61cd54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd13a563edca4cbb838ad02d9d61cd54; no voters:
I20260925 22:45:20.756172 23765 leader_election.cc:288] T 00000000000000000000000000000000 P dd13a563edca4cbb838ad02d9d61cd54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:20.756227 23770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd13a563edca4cbb838ad02d9d61cd54 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:20.756477 23770 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd13a563edca4cbb838ad02d9d61cd54 [term 1 LEADER]: Becoming Leader. State: Replica: dd13a563edca4cbb838ad02d9d61cd54, State: Running, Role: LEADER
I20260925 22:45:20.756861 23770 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd13a563edca4cbb838ad02d9d61cd54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd13a563edca4cbb838ad02d9d61cd54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36819 } }
I20260925 22:45:20.757659 23765 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd13a563edca4cbb838ad02d9d61cd54 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:20.758755 23771 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd13a563edca4cbb838ad02d9d61cd54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd13a563edca4cbb838ad02d9d61cd54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd13a563edca4cbb838ad02d9d61cd54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36819 } } }
I20260925 22:45:20.758913 23771 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd13a563edca4cbb838ad02d9d61cd54 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:20.759184 23771 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd13a563edca4cbb838ad02d9d61cd54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd13a563edca4cbb838ad02d9d61cd54. Latest consensus state: current_term: 1 leader_uuid: "dd13a563edca4cbb838ad02d9d61cd54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd13a563edca4cbb838ad02d9d61cd54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36819 } } }
I20260925 22:45:20.759264 23771 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd13a563edca4cbb838ad02d9d61cd54 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:20.759413 23784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:20.763005 23784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:20.763135 23768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:20.758259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:40261 (local address 127.0.0.1:36819)
0925 22:45:20.758383 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:20.758388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:20.758404 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:45:20.758437 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:20.758439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:20.758506 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:20.758585 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:20.759050 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:20.759882 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:20.761071 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:20.761411 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:20.761524 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:20.762520 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:20.762524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:20.762527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:20.762543 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:20.762575 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:20.762589 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:20.762678 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:20.762861 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 22:45:20.762903 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":57}
I20260925 22:45:20.769786 23784 catalog_manager.cc:1347] Generated new cluster ID: 10816074469f418dbdb94daebc8efc5e
I20260925 22:45:20.769878 23784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:20.787882 23784 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:20.789304 23784 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:20.795830 23784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd13a563edca4cbb838ad02d9d61cd54: Generated new TSK 0
I20260925 22:45:20.796797 23784 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:45:20.803023 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 23697
W20260925 22:45:20.808362 23682 connection.cc:535] client connection to 127.0.0.1:36819 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36819): Connection reset by peer (error 104)
2026-09-25T22:45:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1448 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:20 dist-test-slave-csnl krb5kdc[23796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:20 dist-test-slave-csnl krb5kdc[23796](info): set up 2 sockets
Sep 25 22:45:20 dist-test-slave-csnl krb5kdc[23796](info): commencing operation
krb5kdc: starting...
W20260925 22:45:21.852527 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:45:21 dist-test-slave-csnl krb5kdc[23796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:45:21.984622 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36113
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:22.104357 23809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:22.104755 23809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:22.104868 23809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:22.109390 23809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:22.109517 23809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:22.109556 23809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:22.109597 23809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:22.109642 23809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:22.115595 23809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36113
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376047580784-2347-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:22.116879 23809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:22.118079 23809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:45:22.127861 23809 server_base.cc:1034] running on GCE node
W20260925 22:45:22.127800 23817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:22.127827 23814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:22.127800 23815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:22.128847 23809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:22.130163 23809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:22.131383 23809 hybrid_clock.cc:648] HybridClock initialized: now 1790376322131346 us; error 55 us; skew 500 ppm
Sep 25 22:45:22 dist-test-slave-csnl krb5kdc[23796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:22.134305 23809 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:45:22.135355 23809 webserver.cc:460] Webserver started at http://127.0.0.1:39411/ using document root <none> and password file <none>
I20260925 22:45:22.135962 23809 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:22.136054 23809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:22.136291 23809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:22.138518 23809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "6c7d18dba5b44005b0e8510f6967d0cf"
format_stamp: "Formatted at 2026-09-25 22:45:22 on dist-test-slave-csnl"
server_key: "16b43ba0bf7429b1d179e5e01d70c45e"
server_key_iv: "b8064d1d413999c9e490ba4273b309e4"
server_key_version: "encryptionkey@0"
I20260925 22:45:22.139189 23809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "6c7d18dba5b44005b0e8510f6967d0cf"
format_stamp: "Formatted at 2026-09-25 22:45:22 on dist-test-slave-csnl"
server_key: "16b43ba0bf7429b1d179e5e01d70c45e"
server_key_iv: "b8064d1d413999c9e490ba4273b309e4"
server_key_version: "encryptionkey@0"
I20260925 22:45:22.143298 23809 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:45:22.145910 23824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:22.147459 23809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:22.147590 23809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "6c7d18dba5b44005b0e8510f6967d0cf"
format_stamp: "Formatted at 2026-09-25 22:45:22 on dist-test-slave-csnl"
server_key: "16b43ba0bf7429b1d179e5e01d70c45e"
server_key_iv: "b8064d1d413999c9e490ba4273b309e4"
server_key_version: "encryptionkey@0"
I20260925 22:45:22.147696 23809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:22.152616 23809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:22.155705 23809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:22.155867 23809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:22.163605 23809 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36113
I20260925 22:45:22.163632 23876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36113 every 8 connection(s)
I20260925 22:45:22.164678 23809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:22.167688 23877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:22.172250 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 23809
I20260925 22:45:22.172401 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:45:22.172662 2347 external_mini_cluster.cc:1428] Setting key 3c9e118a955e039bfb53cfca375aee74
I20260925 22:45:22.174052 23877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c7d18dba5b44005b0e8510f6967d0cf: Bootstrap starting.
I20260925 22:45:22.176637 23877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c7d18dba5b44005b0e8510f6967d0cf: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:22.177423 23877 log.cc:824] T 00000000000000000000000000000000 P 6c7d18dba5b44005b0e8510f6967d0cf: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:22.178346 23880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:22.174164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:48197 (local address 127.0.0.1:36113)
0925 22:45:22.174639 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:22.174654 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:22.174693 (+ 39us) server_negotiation.cc:407] Connection header received
0925 22:45:22.175476 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:22.175496 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:22.175814 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:22.176179 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:22.177053 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:22.177058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:22.177070 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:22.177081 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:22.177135 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:22.177156 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:22.177292 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:22.177399 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 22:45:22.177588 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":141,"threads_started":1}
I20260925 22:45:22.179682 23877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c7d18dba5b44005b0e8510f6967d0cf: No bootstrap required, opened a new log
I20260925 22:45:22.181457 2347 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:45:22.184044 23877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c7d18dba5b44005b0e8510f6967d0cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7d18dba5b44005b0e8510f6967d0cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36113 } }
I20260925 22:45:22.184362 23877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c7d18dba5b44005b0e8510f6967d0cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:22.184450 23877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c7d18dba5b44005b0e8510f6967d0cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c7d18dba5b44005b0e8510f6967d0cf, State: Initialized, Role: FOLLOWER
I20260925 22:45:22.184967 23877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c7d18dba5b44005b0e8510f6967d0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7d18dba5b44005b0e8510f6967d0cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36113 } }
I20260925 22:45:22.185146 23877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c7d18dba5b44005b0e8510f6967d0cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:22.185226 23877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c7d18dba5b44005b0e8510f6967d0cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:22.185331 23877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c7d18dba5b44005b0e8510f6967d0cf [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:22.185387 23880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:22.183640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:59411 (local address 127.0.0.1:36113)
0925 22:45:22.183777 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:22.183781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:22.183916 (+ 135us) server_negotiation.cc:407] Connection header received
0925 22:45:22.184019 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:22.184023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:22.184074 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:22.184167 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:22.184927 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:22.184930 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:22.184933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:22.184944 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:22.184974 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:22.184991 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:22.185091 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:22.185214 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 22:45:22.185264 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:45:22.186565 23877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c7d18dba5b44005b0e8510f6967d0cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7d18dba5b44005b0e8510f6967d0cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36113 } }
I20260925 22:45:22.186928 23877 leader_election.cc:302] T 00000000000000000000000000000000 P 6c7d18dba5b44005b0e8510f6967d0cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c7d18dba5b44005b0e8510f6967d0cf; no voters:
I20260925 22:45:22.187284 23877 leader_election.cc:288] T 00000000000000000000000000000000 P 6c7d18dba5b44005b0e8510f6967d0cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:22.187408 23887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c7d18dba5b44005b0e8510f6967d0cf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:22.187635 23887 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c7d18dba5b44005b0e8510f6967d0cf [term 1 LEADER]: Becoming Leader. State: Replica: 6c7d18dba5b44005b0e8510f6967d0cf, State: Running, Role: LEADER
I20260925 22:45:22.187960 23887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c7d18dba5b44005b0e8510f6967d0cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7d18dba5b44005b0e8510f6967d0cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36113 } }
I20260925 22:45:22.188544 23877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c7d18dba5b44005b0e8510f6967d0cf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:22.189155 23887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c7d18dba5b44005b0e8510f6967d0cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c7d18dba5b44005b0e8510f6967d0cf. Latest consensus state: current_term: 1 leader_uuid: "6c7d18dba5b44005b0e8510f6967d0cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7d18dba5b44005b0e8510f6967d0cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36113 } } }
I20260925 22:45:22.189371 23887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c7d18dba5b44005b0e8510f6967d0cf [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:22.189610 23889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c7d18dba5b44005b0e8510f6967d0cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c7d18dba5b44005b0e8510f6967d0cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7d18dba5b44005b0e8510f6967d0cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36113 } } }
I20260925 22:45:22.189744 23889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c7d18dba5b44005b0e8510f6967d0cf [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:22.189922 23895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:22.193398 23895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:22.199471 23895 catalog_manager.cc:1347] Generated new cluster ID: 84495d6f778445109ae44c9398598001
I20260925 22:45:22.199572 23895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:22.230983 23895 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:22.231352 23895 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:22.244930 23895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c7d18dba5b44005b0e8510f6967d0cf: Generated new TSK 0
I20260925 22:45:22.245790 23895 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:45:22.254413 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 23809
2026-09-25T22:45:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1451 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:22 dist-test-slave-csnl krb5kdc[23908](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:22 dist-test-slave-csnl krb5kdc[23908](info): set up 2 sockets
Sep 25 22:45:22 dist-test-slave-csnl krb5kdc[23908](info): commencing operation
krb5kdc: starting...
W20260925 22:45:23.295723 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:45:23 dist-test-slave-csnl krb5kdc[23908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:45:23.423874 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:33327
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:33327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:23.538584 23921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:23.538915 23921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:23.539032 23921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:23.543440 23921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:23.543526 23921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:23.543545 23921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:23.543563 23921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:23.543579 23921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:23.549414 23921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:33327
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:33327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:23.550647 23921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:23.551755 23921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:23.560765 23926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:23.560765 23929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:23.560782 23927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:23.561192 23921 server_base.cc:1034] running on GCE node
I20260925 22:45:23.561794 23921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:23.563077 23921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:23.564258 23921 hybrid_clock.cc:648] HybridClock initialized: now 1790376323564232 us; error 40 us; skew 500 ppm
Sep 25 22:45:23 dist-test-slave-csnl krb5kdc[23908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376323, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:23.572947 23921 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:45:23.574070 23921 webserver.cc:460] Webserver started at http://127.2.74.254:39565/ using document root <none> and password file <none>
I20260925 22:45:23.574801 23921 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:23.574898 23921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:23.575150 23921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:23.577322 23921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "12379d49b02947879f3318f8a00affc8"
format_stamp: "Formatted at 2026-09-25 22:45:23 on dist-test-slave-csnl"
server_key: "1a8c59cccb9889c70b4e7be48f6647d0"
server_key_iv: "3f6b555b96c655b51d8dac079fbf366f"
server_key_version: "encryptionkey@0"
I20260925 22:45:23.577946 23921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "12379d49b02947879f3318f8a00affc8"
format_stamp: "Formatted at 2026-09-25 22:45:23 on dist-test-slave-csnl"
server_key: "1a8c59cccb9889c70b4e7be48f6647d0"
server_key_iv: "3f6b555b96c655b51d8dac079fbf366f"
server_key_version: "encryptionkey@0"
I20260925 22:45:23.581981 23921 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 22:45:23.584780 23936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:23.586097 23921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:45:23.586253 23921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "12379d49b02947879f3318f8a00affc8"
format_stamp: "Formatted at 2026-09-25 22:45:23 on dist-test-slave-csnl"
server_key: "1a8c59cccb9889c70b4e7be48f6647d0"
server_key_iv: "3f6b555b96c655b51d8dac079fbf366f"
server_key_version: "encryptionkey@0"
I20260925 22:45:23.586390 23921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:23.607137 23921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:23.610448 23921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:23.610682 23921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:23.618146 23921 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:33327
I20260925 22:45:23.618170 23988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:33327 every 8 connection(s)
I20260925 22:45:23.619233 23921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:23.621320 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 23921
I20260925 22:45:23.621481 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:45:23.621780 2347 external_mini_cluster.cc:1428] Setting key 30a673e6e1b2a3ed216451cea54c6dfa
I20260925 22:45:23.622769 23989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:23.629287 23989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12379d49b02947879f3318f8a00affc8: Bootstrap starting.
Sep 25 22:45:23 dist-test-slave-csnl krb5kdc[23908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:45:23.632176 23989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12379d49b02947879f3318f8a00affc8: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:23.633023 23989 log.cc:824] T 00000000000000000000000000000000 P 12379d49b02947879f3318f8a00affc8: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:23.635321 23989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12379d49b02947879f3318f8a00affc8: No bootstrap required, opened a new log
I20260925 22:45:23.637459 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:23.623451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34283 (local address 127.2.74.254:33327)
0925 22:45:23.623979 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:23.623996 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:23.624035 (+ 39us) server_negotiation.cc:407] Connection header received
0925 22:45:23.624735 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:23.624754 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:23.625173 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:23.625560 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:23.626492 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.627603 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.628515 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.629033 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.629205 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:23.632195 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:23.632202 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:23.632218 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:23.632279 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:23.634151 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:23.634796 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:23.634802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:23.634807 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:23.634935 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:23.635342 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:23.635346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:23.635348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:23.635826 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:23.636072 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:23.636376 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 22:45:23.636627 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":180,"threads_started":1}
I20260925 22:45:23.639634 23989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12379d49b02947879f3318f8a00affc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12379d49b02947879f3318f8a00affc8" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33327 } }
I20260925 22:45:23.639992 23989 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12379d49b02947879f3318f8a00affc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:23.640079 23989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12379d49b02947879f3318f8a00affc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12379d49b02947879f3318f8a00affc8, State: Initialized, Role: FOLLOWER
I20260925 22:45:23.640637 23989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12379d49b02947879f3318f8a00affc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12379d49b02947879f3318f8a00affc8" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33327 } }
I20260925 22:45:23.640820 23989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12379d49b02947879f3318f8a00affc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:23.640887 23989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12379d49b02947879f3318f8a00affc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:23.641005 23989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12379d49b02947879f3318f8a00affc8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:23.641324 2347 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:45:23.642202 23989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12379d49b02947879f3318f8a00affc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12379d49b02947879f3318f8a00affc8" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33327 } }
I20260925 22:45:23.642638 23989 leader_election.cc:302] T 00000000000000000000000000000000 P 12379d49b02947879f3318f8a00affc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12379d49b02947879f3318f8a00affc8; no voters:
I20260925 22:45:23.643062 23989 leader_election.cc:288] T 00000000000000000000000000000000 P 12379d49b02947879f3318f8a00affc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:23.643168 23998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12379d49b02947879f3318f8a00affc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:23.643416 23998 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12379d49b02947879f3318f8a00affc8 [term 1 LEADER]: Becoming Leader. State: Replica: 12379d49b02947879f3318f8a00affc8, State: Running, Role: LEADER
I20260925 22:45:23.643865 23998 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12379d49b02947879f3318f8a00affc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12379d49b02947879f3318f8a00affc8" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33327 } }
I20260925 22:45:23.644482 23989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12379d49b02947879f3318f8a00affc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:23.645084 23998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12379d49b02947879f3318f8a00affc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12379d49b02947879f3318f8a00affc8. Latest consensus state: current_term: 1 leader_uuid: "12379d49b02947879f3318f8a00affc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12379d49b02947879f3318f8a00affc8" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33327 } } }
I20260925 22:45:23.645222 23998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12379d49b02947879f3318f8a00affc8 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:23.645514 23999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12379d49b02947879f3318f8a00affc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12379d49b02947879f3318f8a00affc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12379d49b02947879f3318f8a00affc8" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 33327 } } }
I20260925 22:45:23.645671 23999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12379d49b02947879f3318f8a00affc8 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:23.646028 24005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:23.649369 24005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:23.656072 24005 catalog_manager.cc:1347] Generated new cluster ID: b40d4c5050724428bf93a79800156f61
I20260925 22:45:23.656443 24005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:23.658818 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:23.643650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32819 (local address 127.2.74.254:33327)
0925 22:45:23.643771 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:23.643775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:23.645974 (+ 2199us) server_negotiation.cc:407] Connection header received
0925 22:45:23.646027 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:23.646029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:23.646086 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:23.646162 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:23.647194 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.647970 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.650015 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.650395 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.650483 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:23.655138 (+ 4655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:23.655142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:23.655145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:23.655179 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:23.656938 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:23.657515 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:23.657518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:23.657520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:23.657596 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:23.657927 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:23.657931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:23.657933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:23.658115 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:23.658271 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:23.658544 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 22:45:23.658646 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:45:23.663280 24005 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:23.664254 24005 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:23.672900 24005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12379d49b02947879f3318f8a00affc8: Generated new TSK 0
I20260925 22:45:23.673749 24005 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:45:23.682286 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 23921
W20260925 22:45:23.687394 23906 connection.cc:535] client connection to 127.2.74.254:33327 recv error: Network error: failed to read from TLS socket (remote: 127.2.74.254:33327): Connection reset by peer (error 104)
2026-09-25T22:45:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1427 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:23 dist-test-slave-csnl krb5kdc[24020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:23 dist-test-slave-csnl krb5kdc[24020](info): set up 2 sockets
Sep 25 22:45:23 dist-test-slave-csnl krb5kdc[24020](info): commencing operation
krb5kdc: starting...
W20260925 22:45:24.711335 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:45:24 dist-test-slave-csnl krb5kdc[24020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:45:24.834041 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:40637
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:36003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:40637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:24.955181 24033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:24.955469 24033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:24.955514 24033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:24.959728 24033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:24.959808 24033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:24.959828 24033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:24.959846 24033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:24.959860 24033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:24.965516 24033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:36003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:40637
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:40637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:24.966706 24033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:24.967818 24033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:24.976863 24039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:24.976848 24041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:24.976848 24038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:24.977046 24033 server_base.cc:1034] running on GCE node
I20260925 22:45:24.977876 24033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:24.979104 24033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:24.980314 24033 hybrid_clock.cc:648] HybridClock initialized: now 1790376324980276 us; error 53 us; skew 500 ppm
Sep 25 22:45:24 dist-test-slave-csnl krb5kdc[24020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376324, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:24.983351 24033 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:45:24.984418 24033 webserver.cc:460] Webserver started at http://127.2.74.254:35145/ using document root <none> and password file <none>
I20260925 22:45:24.985059 24033 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:24.985164 24033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:24.985407 24033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:24.987749 24033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "198fb244eb404646a181e4fa9f92a327"
format_stamp: "Formatted at 2026-09-25 22:45:24 on dist-test-slave-csnl"
server_key: "c9725c947720d54c396bdb60772cd144"
server_key_iv: "10c8fbf60c2fdf794b0ced88257774d0"
server_key_version: "encryptionkey@0"
I20260925 22:45:24.988353 24033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "198fb244eb404646a181e4fa9f92a327"
format_stamp: "Formatted at 2026-09-25 22:45:24 on dist-test-slave-csnl"
server_key: "c9725c947720d54c396bdb60772cd144"
server_key_iv: "10c8fbf60c2fdf794b0ced88257774d0"
server_key_version: "encryptionkey@0"
I20260925 22:45:24.992475 24033 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:45:24.995386 24048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:24.996802 24033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:24.997002 24033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "198fb244eb404646a181e4fa9f92a327"
format_stamp: "Formatted at 2026-09-25 22:45:24 on dist-test-slave-csnl"
server_key: "c9725c947720d54c396bdb60772cd144"
server_key_iv: "10c8fbf60c2fdf794b0ced88257774d0"
server_key_version: "encryptionkey@0"
I20260925 22:45:24.997150 24033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:25.015759 24033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:25.019518 24033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:25.019775 24033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:25.027634 24033 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:40637
I20260925 22:45:25.027647 24100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:40637 every 8 connection(s)
I20260925 22:45:25.028693 24033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:25.031713 24101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:25.031950 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 24033
I20260925 22:45:25.032114 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:45:25.032418 2347 external_mini_cluster.cc:1428] Setting key e35876be5d0aff661341f14a5d06fb6e
I20260925 22:45:25.038642 24101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 198fb244eb404646a181e4fa9f92a327: Bootstrap starting.
I20260925 22:45:25.041338 24101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 198fb244eb404646a181e4fa9f92a327: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:25.042161 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:25.033793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35179 (local address 127.2.74.254:40637)
0925 22:45:25.034270 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:25.034285 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:25.034332 (+ 47us) server_negotiation.cc:407] Connection header received
0925 22:45:25.035256 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:25.035279 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:25.035653 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:25.035999 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:25.036418 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:25.037442 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:25.038383 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:25.038996 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:25.039181 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:25.040255 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:25.040260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:25.040275 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:25.040713 (+ 438us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:25.040779 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:25.040804 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:25.040957 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:25.041164 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 22:45:25.041318 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":154,"threads_started":1}
I20260925 22:45:25.042194 24101 log.cc:824] T 00000000000000000000000000000000 P 198fb244eb404646a181e4fa9f92a327: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:25.044790 24101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 198fb244eb404646a181e4fa9f92a327: No bootstrap required, opened a new log
I20260925 22:45:25.045771 2347 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:45:25.049309 24101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 198fb244eb404646a181e4fa9f92a327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198fb244eb404646a181e4fa9f92a327" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40637 } }
I20260925 22:45:25.049643 24101 raft_consensus.cc:374] T 00000000000000000000000000000000 P 198fb244eb404646a181e4fa9f92a327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:25.049711 24101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 198fb244eb404646a181e4fa9f92a327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 198fb244eb404646a181e4fa9f92a327, State: Initialized, Role: FOLLOWER
I20260925 22:45:25.050231 24101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 198fb244eb404646a181e4fa9f92a327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198fb244eb404646a181e4fa9f92a327" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40637 } }
I20260925 22:45:25.050405 24101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 198fb244eb404646a181e4fa9f92a327 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:25.050487 24101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 198fb244eb404646a181e4fa9f92a327 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:25.050595 24101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 198fb244eb404646a181e4fa9f92a327 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:25.052023 24101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 198fb244eb404646a181e4fa9f92a327 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198fb244eb404646a181e4fa9f92a327" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40637 } }
I20260925 22:45:25.052459 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:25.047995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38609 (local address 127.2.74.254:40637)
0925 22:45:25.048148 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:25.048154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:25.048352 (+ 198us) server_negotiation.cc:407] Connection header received
0925 22:45:25.048523 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:25.048527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:25.048572 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:25.048656 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:25.048949 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:25.049524 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:25.050457 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:25.050853 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:25.051036 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:25.051920 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:25.051923 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:25.051928 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:25.051949 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:25.051984 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:25.052001 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:25.052100 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:25.052283 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 22:45:25.052325 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:45:25.052491 24101 leader_election.cc:302] T 00000000000000000000000000000000 P 198fb244eb404646a181e4fa9f92a327 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 198fb244eb404646a181e4fa9f92a327; no voters:
I20260925 22:45:25.052897 24101 leader_election.cc:288] T 00000000000000000000000000000000 P 198fb244eb404646a181e4fa9f92a327 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:25.053018 24111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 198fb244eb404646a181e4fa9f92a327 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:25.053248 24111 raft_consensus.cc:686] T 00000000000000000000000000000000 P 198fb244eb404646a181e4fa9f92a327 [term 1 LEADER]: Becoming Leader. State: Replica: 198fb244eb404646a181e4fa9f92a327, State: Running, Role: LEADER
I20260925 22:45:25.053649 24111 consensus_queue.cc:237] T 00000000000000000000000000000000 P 198fb244eb404646a181e4fa9f92a327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198fb244eb404646a181e4fa9f92a327" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40637 } }
I20260925 22:45:25.054402 24101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 198fb244eb404646a181e4fa9f92a327 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:25.054927 24111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198fb244eb404646a181e4fa9f92a327 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 198fb244eb404646a181e4fa9f92a327. Latest consensus state: current_term: 1 leader_uuid: "198fb244eb404646a181e4fa9f92a327" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198fb244eb404646a181e4fa9f92a327" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40637 } } }
I20260925 22:45:25.055176 24111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198fb244eb404646a181e4fa9f92a327 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:25.055437 24112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198fb244eb404646a181e4fa9f92a327 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "198fb244eb404646a181e4fa9f92a327" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198fb244eb404646a181e4fa9f92a327" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40637 } } }
I20260925 22:45:25.055629 24112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198fb244eb404646a181e4fa9f92a327 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:25.055639 24116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:25.059244 24116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:25.066273 24116 catalog_manager.cc:1347] Generated new cluster ID: f0c6febc1003473eb0f9eb76f3336aaa
I20260925 22:45:25.066383 24116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:25.076869 24116 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:25.078262 24116 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:25.092168 24116 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 198fb244eb404646a181e4fa9f92a327: Generated new TSK 0
I20260925 22:45:25.093011 24116 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:45:25.100158 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 24033
W20260925 22:45:25.106000 24018 connection.cc:535] client connection to 127.2.74.254:40637 recv error: Network error: failed to read from TLS socket (remote: 127.2.74.254:40637): Connection reset by peer (error 104)
2026-09-25T22:45:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:25 dist-test-slave-csnl krb5kdc[24132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:25 dist-test-slave-csnl krb5kdc[24132](info): set up 2 sockets
Sep 25 22:45:25 dist-test-slave-csnl krb5kdc[24132](info): commencing operation
krb5kdc: starting...
W20260925 22:45:26.135450 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:45:26 dist-test-slave-csnl krb5kdc[24132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:45:26.261929 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:34289
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:37997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:34289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:26.377027 24145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:26.377422 24145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:26.377517 24145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:26.382055 24145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:26.382165 24145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:26.382221 24145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:26.382272 24145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:26.382310 24145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:26.388238 24145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:37997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:34289
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:34289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:26.389473 24145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:26.390563 24145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:26.399552 24150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:26.399593 24151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:26.399556 24153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:26.399812 24145 server_base.cc:1034] running on GCE node
I20260925 22:45:26.400466 24145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:26.401664 24145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:26.402840 24145 hybrid_clock.cc:648] HybridClock initialized: now 1790376326402814 us; error 41 us; skew 500 ppm
Sep 25 22:45:26 dist-test-slave-csnl krb5kdc[24132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376326, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:26.405642 24145 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:45:26.406591 24145 webserver.cc:460] Webserver started at http://127.2.74.254:38979/ using document root <none> and password file <none>
I20260925 22:45:26.407202 24145 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:26.407271 24145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:26.407445 24145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:26.409701 24145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "736fc9266a064f02aa3c8b5700a55da7"
format_stamp: "Formatted at 2026-09-25 22:45:26 on dist-test-slave-csnl"
server_key: "eb17112627d1bdedbf32fb760d579aa1"
server_key_iv: "078363f25128906445f9beca548c127a"
server_key_version: "encryptionkey@0"
I20260925 22:45:26.410285 24145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "736fc9266a064f02aa3c8b5700a55da7"
format_stamp: "Formatted at 2026-09-25 22:45:26 on dist-test-slave-csnl"
server_key: "eb17112627d1bdedbf32fb760d579aa1"
server_key_iv: "078363f25128906445f9beca548c127a"
server_key_version: "encryptionkey@0"
I20260925 22:45:26.414358 24145 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:45:26.416994 24160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:26.418464 24145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:45:26.418613 24145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "736fc9266a064f02aa3c8b5700a55da7"
format_stamp: "Formatted at 2026-09-25 22:45:26 on dist-test-slave-csnl"
server_key: "eb17112627d1bdedbf32fb760d579aa1"
server_key_iv: "078363f25128906445f9beca548c127a"
server_key_version: "encryptionkey@0"
I20260925 22:45:26.418745 24145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:26.425047 24145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:26.428364 24145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:26.428583 24145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:26.436089 24145 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:34289
I20260925 22:45:26.436106 24212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:34289 every 8 connection(s)
I20260925 22:45:26.437098 24145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:26.438625 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 24145
I20260925 22:45:26.438776 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:45:26.439144 2347 external_mini_cluster.cc:1428] Setting key c13d3b0c0dfb97c79518d15c277db08b
I20260925 22:45:26.440227 24213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:26.444584 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:26.440609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46445 (local address 127.2.74.254:34289)
0925 22:45:26.441068 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:26.441084 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:26.441121 (+ 37us) server_negotiation.cc:407] Connection header received
0925 22:45:26.441787 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:26.441805 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:26.442104 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:26.442469 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:26.443336 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:26.443341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:26.443355 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:26.443365 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:26.443422 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:26.443442 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:26.443596 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:26.443670 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 22:45:26.443838 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":137,"threads_started":1}
I20260925 22:45:26.446976 24213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 736fc9266a064f02aa3c8b5700a55da7: Bootstrap starting.
I20260925 22:45:26.447559 2347 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:45:26.449442 24213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 736fc9266a064f02aa3c8b5700a55da7: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:26.450467 24213 log.cc:824] T 00000000000000000000000000000000 P 736fc9266a064f02aa3c8b5700a55da7: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:26.451344 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:26.449599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54195 (local address 127.2.74.254:34289)
0925 22:45:26.449752 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:26.449758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:26.449925 (+ 167us) server_negotiation.cc:407] Connection header received
0925 22:45:26.450031 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:26.450034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:26.450072 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:26.450162 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:26.450885 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:26.450892 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:26.450898 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:26.450914 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:26.450967 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:26.450986 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:26.451066 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:26.451160 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 22:45:26.451194 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:45:26.452916 24213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 736fc9266a064f02aa3c8b5700a55da7: No bootstrap required, opened a new log
I20260925 22:45:26.456707 24213 raft_consensus.cc:348] T 00000000000000000000000000000000 P 736fc9266a064f02aa3c8b5700a55da7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736fc9266a064f02aa3c8b5700a55da7" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34289 } }
I20260925 22:45:26.457011 24213 raft_consensus.cc:374] T 00000000000000000000000000000000 P 736fc9266a064f02aa3c8b5700a55da7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:26.457075 24213 raft_consensus.cc:729] T 00000000000000000000000000000000 P 736fc9266a064f02aa3c8b5700a55da7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 736fc9266a064f02aa3c8b5700a55da7, State: Initialized, Role: FOLLOWER
I20260925 22:45:26.457589 24213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 736fc9266a064f02aa3c8b5700a55da7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736fc9266a064f02aa3c8b5700a55da7" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34289 } }
I20260925 22:45:26.457752 24213 raft_consensus.cc:388] T 00000000000000000000000000000000 P 736fc9266a064f02aa3c8b5700a55da7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:26.457840 24213 raft_consensus.cc:482] T 00000000000000000000000000000000 P 736fc9266a064f02aa3c8b5700a55da7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:26.457950 24213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 736fc9266a064f02aa3c8b5700a55da7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:26.459175 24213 raft_consensus.cc:504] T 00000000000000000000000000000000 P 736fc9266a064f02aa3c8b5700a55da7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736fc9266a064f02aa3c8b5700a55da7" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34289 } }
I20260925 22:45:26.459528 24213 leader_election.cc:302] T 00000000000000000000000000000000 P 736fc9266a064f02aa3c8b5700a55da7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 736fc9266a064f02aa3c8b5700a55da7; no voters:
I20260925 22:45:26.459821 24213 leader_election.cc:288] T 00000000000000000000000000000000 P 736fc9266a064f02aa3c8b5700a55da7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:26.459980 24223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 736fc9266a064f02aa3c8b5700a55da7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:26.460196 24223 raft_consensus.cc:686] T 00000000000000000000000000000000 P 736fc9266a064f02aa3c8b5700a55da7 [term 1 LEADER]: Becoming Leader. State: Replica: 736fc9266a064f02aa3c8b5700a55da7, State: Running, Role: LEADER
I20260925 22:45:26.460556 24223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 736fc9266a064f02aa3c8b5700a55da7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736fc9266a064f02aa3c8b5700a55da7" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34289 } }
I20260925 22:45:26.461160 24213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 736fc9266a064f02aa3c8b5700a55da7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:26.462276 24225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 736fc9266a064f02aa3c8b5700a55da7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 736fc9266a064f02aa3c8b5700a55da7. Latest consensus state: current_term: 1 leader_uuid: "736fc9266a064f02aa3c8b5700a55da7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736fc9266a064f02aa3c8b5700a55da7" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34289 } } }
I20260925 22:45:26.462414 24225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 736fc9266a064f02aa3c8b5700a55da7 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:26.462752 24224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 736fc9266a064f02aa3c8b5700a55da7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "736fc9266a064f02aa3c8b5700a55da7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736fc9266a064f02aa3c8b5700a55da7" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34289 } } }
I20260925 22:45:26.462882 24224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 736fc9266a064f02aa3c8b5700a55da7 [sys.catalog]: This master's current role is: LEADER
W20260925 22:45:26.466243 24238 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 736fc9266a064f02aa3c8b5700a55da7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:45:26.466328 24238 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:45:26.466416 24232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:26.467222 24232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:26.473493 24232 catalog_manager.cc:1347] Generated new cluster ID: a0f9c09e9a384398900b691affd19695
I20260925 22:45:26.473587 24232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:26.489349 24232 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:26.489786 24232 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:26.500603 24232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 736fc9266a064f02aa3c8b5700a55da7: Generated new TSK 0
I20260925 22:45:26.501425 24232 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:45:26.511292 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 24145
2026-09-25T22:45:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:26 dist-test-slave-csnl krb5kdc[24244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:26 dist-test-slave-csnl krb5kdc[24244](info): set up 2 sockets
Sep 25 22:45:26 dist-test-slave-csnl krb5kdc[24244](info): commencing operation
krb5kdc: starting...
W20260925 22:45:27.539397 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:45:27 dist-test-slave-csnl krb5kdc[24244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:45:27.665558 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:34673
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:41011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:34673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:27.779204 24257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:27.779572 24257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:27.779666 24257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:27.783914 24257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:27.784035 24257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:27.784073 24257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:27.784109 24257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:27.784138 24257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:27.789875 24257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:41011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:34673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:34673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:27.791460 24257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:27.792757 24257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:27.802260 24262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:27.802260 24263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:27.802270 24265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:27.802708 24257 server_base.cc:1034] running on GCE node
I20260925 22:45:27.803413 24257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:27.804731 24257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:27.805946 24257 hybrid_clock.cc:648] HybridClock initialized: now 1790376327805924 us; error 37 us; skew 500 ppm
Sep 25 22:45:27 dist-test-slave-csnl krb5kdc[24244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376327, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:27.809154 24257 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:45:27.810241 24257 webserver.cc:460] Webserver started at http://127.2.74.254:46387/ using document root <none> and password file <none>
I20260925 22:45:27.810850 24257 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:27.810925 24257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:27.811192 24257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:27.813541 24257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "6a395cf27c4a4d2d8d37e64e179e6f09"
format_stamp: "Formatted at 2026-09-25 22:45:27 on dist-test-slave-csnl"
server_key: "0a74e1a9ac40af37d744840a78b80b36"
server_key_iv: "1bca0d302f0ee7c533165d5de33f14cb"
server_key_version: "encryptionkey@0"
I20260925 22:45:27.814143 24257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "6a395cf27c4a4d2d8d37e64e179e6f09"
format_stamp: "Formatted at 2026-09-25 22:45:27 on dist-test-slave-csnl"
server_key: "0a74e1a9ac40af37d744840a78b80b36"
server_key_iv: "1bca0d302f0ee7c533165d5de33f14cb"
server_key_version: "encryptionkey@0"
I20260925 22:45:27.818435 24257 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:45:27.821559 24272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:27.823033 24257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 22:45:27.823190 24257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "6a395cf27c4a4d2d8d37e64e179e6f09"
format_stamp: "Formatted at 2026-09-25 22:45:27 on dist-test-slave-csnl"
server_key: "0a74e1a9ac40af37d744840a78b80b36"
server_key_iv: "1bca0d302f0ee7c533165d5de33f14cb"
server_key_version: "encryptionkey@0"
I20260925 22:45:27.823323 24257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:27.838706 24257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:27.842113 24257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:27.842471 24257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:27.850687 24257 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:34673
I20260925 22:45:27.850708 24324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:34673 every 8 connection(s)
I20260925 22:45:27.851856 24257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:27.852305 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 24257
I20260925 22:45:27.852454 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:45:27.852761 2347 external_mini_cluster.cc:1428] Setting key 205ecb83866a851dfd6eae205292211c
I20260925 22:45:27.855683 24325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 22:45:27 dist-test-slave-csnl krb5kdc[24244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:45:27.863404 24325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a395cf27c4a4d2d8d37e64e179e6f09: Bootstrap starting.
I20260925 22:45:27.866703 24325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a395cf27c4a4d2d8d37e64e179e6f09: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:27.867635 24325 log.cc:824] T 00000000000000000000000000000000 P 6a395cf27c4a4d2d8d37e64e179e6f09: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:27.869107 24328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:27.854236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55345 (local address 127.2.74.254:34673)
0925 22:45:27.854716 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:27.854738 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:27.854787 (+ 49us) server_negotiation.cc:407] Connection header received
0925 22:45:27.855794 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:27.855822 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:27.856327 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:27.856747 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:27.857495 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:27.859225 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:27.860122 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:27.860928 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:27.861112 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:27.863739 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:27.863746 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:27.863764 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:27.863804 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:27.865947 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:27.866430 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:27.866437 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:27.866444 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:27.866627 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:27.867049 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:27.867052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:27.867053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:27.867495 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:27.867734 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:27.867993 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 22:45:27.868348 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":128,"threads_started":1}
I20260925 22:45:27.869872 24325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a395cf27c4a4d2d8d37e64e179e6f09: No bootstrap required, opened a new log
I20260925 22:45:27.872397 2347 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:45:27.875294 24325 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a395cf27c4a4d2d8d37e64e179e6f09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a395cf27c4a4d2d8d37e64e179e6f09" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34673 } }
I20260925 22:45:27.875597 24325 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a395cf27c4a4d2d8d37e64e179e6f09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:27.875679 24325 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a395cf27c4a4d2d8d37e64e179e6f09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a395cf27c4a4d2d8d37e64e179e6f09, State: Initialized, Role: FOLLOWER
I20260925 22:45:27.876230 24325 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a395cf27c4a4d2d8d37e64e179e6f09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a395cf27c4a4d2d8d37e64e179e6f09" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34673 } }
I20260925 22:45:27.876413 24325 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a395cf27c4a4d2d8d37e64e179e6f09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:27.876506 24325 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a395cf27c4a4d2d8d37e64e179e6f09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:27.876626 24325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a395cf27c4a4d2d8d37e64e179e6f09 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:27.877743 24325 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a395cf27c4a4d2d8d37e64e179e6f09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a395cf27c4a4d2d8d37e64e179e6f09" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34673 } }
I20260925 22:45:27.878196 24325 leader_election.cc:302] T 00000000000000000000000000000000 P 6a395cf27c4a4d2d8d37e64e179e6f09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a395cf27c4a4d2d8d37e64e179e6f09; no voters:
I20260925 22:45:27.878634 24325 leader_election.cc:288] T 00000000000000000000000000000000 P 6a395cf27c4a4d2d8d37e64e179e6f09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:27.878762 24335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a395cf27c4a4d2d8d37e64e179e6f09 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:27.879058 24335 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a395cf27c4a4d2d8d37e64e179e6f09 [term 1 LEADER]: Becoming Leader. State: Replica: 6a395cf27c4a4d2d8d37e64e179e6f09, State: Running, Role: LEADER
I20260925 22:45:27.879496 24335 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a395cf27c4a4d2d8d37e64e179e6f09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a395cf27c4a4d2d8d37e64e179e6f09" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34673 } }
I20260925 22:45:27.880155 24325 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a395cf27c4a4d2d8d37e64e179e6f09 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:27.881861 24336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a395cf27c4a4d2d8d37e64e179e6f09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a395cf27c4a4d2d8d37e64e179e6f09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a395cf27c4a4d2d8d37e64e179e6f09" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34673 } } }
I20260925 22:45:27.882042 24336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a395cf27c4a4d2d8d37e64e179e6f09 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:27.881989 24337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a395cf27c4a4d2d8d37e64e179e6f09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a395cf27c4a4d2d8d37e64e179e6f09. Latest consensus state: current_term: 1 leader_uuid: "6a395cf27c4a4d2d8d37e64e179e6f09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a395cf27c4a4d2d8d37e64e179e6f09" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34673 } } }
I20260925 22:45:27.882145 24337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a395cf27c4a4d2d8d37e64e179e6f09 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:27.884908 24328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:27.874882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33021 (local address 127.2.74.254:34673)
0925 22:45:27.875053 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:27.875058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:27.875252 (+ 194us) server_negotiation.cc:407] Connection header received
0925 22:45:27.875412 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:27.875420 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:27.875464 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:27.875543 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:27.876336 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:27.877117 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:27.877962 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:27.878372 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:27.878518 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:27.880294 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:27.880299 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:27.880302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:27.880336 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:27.882437 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:27.883044 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:27.883048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:27.883049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:27.883141 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:27.883519 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:27.883522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:27.883524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:27.883764 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:27.883902 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:27.884634 (+ 732us) server_negotiation.cc:299] Negotiation successful
0925 22:45:27.884771 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 22:45:27.886286 24350 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6a395cf27c4a4d2d8d37e64e179e6f09: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:45:27.886381 24350 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:45:27.886447 24345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:27.887434 24345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:27.893543 24345 catalog_manager.cc:1347] Generated new cluster ID: 4993793fba1948089839166fec7f3cfa
I20260925 22:45:27.893685 24345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:27.915616 24345 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:27.917142 24345 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:27.925894 24345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a395cf27c4a4d2d8d37e64e179e6f09: Generated new TSK 0
I20260925 22:45:27.926717 24345 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:45:27.937976 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 24257
2026-09-25T22:45:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:27 dist-test-slave-csnl krb5kdc[24356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:27 dist-test-slave-csnl krb5kdc[24356](info): set up 2 sockets
Sep 25 22:45:27 dist-test-slave-csnl krb5kdc[24356](info): commencing operation
krb5kdc: starting...
W20260925 22:45:28.987708 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:45:29 dist-test-slave-csnl krb5kdc[24356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:45:29.114620 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:39041
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:35533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:39041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:29.231388 24369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:29.231861 24369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:29.231943 24369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:29.236338 24369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:29.236413 24369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:29.236434 24369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:29.236451 24369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:29.236488 24369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:29.242049 24369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:35533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:39041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:39041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:29.243328 24369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:29.244432 24369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:29.253862 24377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:29.253862 24375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:29.253856 24374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:29.254354 24369 server_base.cc:1034] running on GCE node
I20260925 22:45:29.255079 24369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:29.256307 24369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:29.257507 24369 hybrid_clock.cc:648] HybridClock initialized: now 1790376329257496 us; error 37 us; skew 500 ppm
Sep 25 22:45:29 dist-test-slave-csnl krb5kdc[24356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376329, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:29.264308 24369 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:45:29.265424 24369 webserver.cc:460] Webserver started at http://127.2.74.254:43159/ using document root <none> and password file <none>
I20260925 22:45:29.266057 24369 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:29.266170 24369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:29.266428 24369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:29.268775 24369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "8ea33b21aafd497d8c9d6ce3546c429b"
format_stamp: "Formatted at 2026-09-25 22:45:29 on dist-test-slave-csnl"
server_key: "a9618b5c45bf2d4b24ba948a707a9e63"
server_key_iv: "0e70d879166f610f61f4685d1dccebc5"
server_key_version: "encryptionkey@0"
I20260925 22:45:29.269412 24369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "8ea33b21aafd497d8c9d6ce3546c429b"
format_stamp: "Formatted at 2026-09-25 22:45:29 on dist-test-slave-csnl"
server_key: "a9618b5c45bf2d4b24ba948a707a9e63"
server_key_iv: "0e70d879166f610f61f4685d1dccebc5"
server_key_version: "encryptionkey@0"
I20260925 22:45:29.273667 24369 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:45:29.276511 24384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:29.277943 24369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:45:29.278111 24369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "8ea33b21aafd497d8c9d6ce3546c429b"
format_stamp: "Formatted at 2026-09-25 22:45:29 on dist-test-slave-csnl"
server_key: "a9618b5c45bf2d4b24ba948a707a9e63"
server_key_iv: "0e70d879166f610f61f4685d1dccebc5"
server_key_version: "encryptionkey@0"
I20260925 22:45:29.278257 24369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:29.294870 24369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:29.298063 24369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:29.298298 24369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:29.306035 24369 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:39041
I20260925 22:45:29.306058 24436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:39041 every 8 connection(s)
I20260925 22:45:29.307171 24369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:29.310141 24437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:29.311738 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 24369
I20260925 22:45:29.311856 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:45:29.312171 2347 external_mini_cluster.cc:1428] Setting key 834ba1766f9507610e90bea05a50b449
I20260925 22:45:29.317384 24437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ea33b21aafd497d8c9d6ce3546c429b: Bootstrap starting.
I20260925 22:45:29.320252 24437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ea33b21aafd497d8c9d6ce3546c429b: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:29.321267 24437 log.cc:824] T 00000000000000000000000000000000 P 8ea33b21aafd497d8c9d6ce3546c429b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:29.322772 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:29.313649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35361 (local address 127.2.74.254:39041)
0925 22:45:29.314120 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:29.314147 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:29.314194 (+ 47us) server_negotiation.cc:407] Connection header received
0925 22:45:29.315120 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:29.315149 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:29.315622 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:29.316032 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:29.316468 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:29.317895 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:29.318883 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:29.319633 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:29.319806 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:29.320678 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:29.320684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:29.320704 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:29.321259 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:29.321335 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:29.321371 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:29.321470 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:29.321671 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 22:45:29.321902 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":127,"threads_started":1}
I20260925 22:45:29.323760 24437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ea33b21aafd497d8c9d6ce3546c429b: No bootstrap required, opened a new log
I20260925 22:45:29.325870 2347 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:45:29.328109 24437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ea33b21aafd497d8c9d6ce3546c429b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea33b21aafd497d8c9d6ce3546c429b" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 39041 } }
I20260925 22:45:29.328464 24437 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ea33b21aafd497d8c9d6ce3546c429b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:29.328569 24437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ea33b21aafd497d8c9d6ce3546c429b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ea33b21aafd497d8c9d6ce3546c429b, State: Initialized, Role: FOLLOWER
I20260925 22:45:29.329088 24437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ea33b21aafd497d8c9d6ce3546c429b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea33b21aafd497d8c9d6ce3546c429b" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 39041 } }
I20260925 22:45:29.329265 24437 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ea33b21aafd497d8c9d6ce3546c429b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:29.329356 24437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ea33b21aafd497d8c9d6ce3546c429b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:29.329466 24437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ea33b21aafd497d8c9d6ce3546c429b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:29.330533 24437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ea33b21aafd497d8c9d6ce3546c429b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea33b21aafd497d8c9d6ce3546c429b" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 39041 } }
I20260925 22:45:29.330928 24437 leader_election.cc:302] T 00000000000000000000000000000000 P 8ea33b21aafd497d8c9d6ce3546c429b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ea33b21aafd497d8c9d6ce3546c429b; no voters:
I20260925 22:45:29.331436 24437 leader_election.cc:288] T 00000000000000000000000000000000 P 8ea33b21aafd497d8c9d6ce3546c429b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:29.331486 24447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ea33b21aafd497d8c9d6ce3546c429b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:29.331717 24447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ea33b21aafd497d8c9d6ce3546c429b [term 1 LEADER]: Becoming Leader. State: Replica: 8ea33b21aafd497d8c9d6ce3546c429b, State: Running, Role: LEADER
I20260925 22:45:29.332108 24447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ea33b21aafd497d8c9d6ce3546c429b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea33b21aafd497d8c9d6ce3546c429b" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 39041 } }
I20260925 22:45:29.332886 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:29.328422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57865 (local address 127.2.74.254:39041)
0925 22:45:29.328577 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:29.328595 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:29.328778 (+ 183us) server_negotiation.cc:407] Connection header received
0925 22:45:29.328940 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:29.328945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:29.328993 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:29.329077 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:29.329372 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:29.330029 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:29.330930 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:29.331301 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:29.331419 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:29.332409 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:29.332412 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:29.332415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:29.332431 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:29.332462 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:29.332474 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:29.332533 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:29.332690 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 22:45:29.332752 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 22:45:29.332914 24437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ea33b21aafd497d8c9d6ce3546c429b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:29.333839 24448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ea33b21aafd497d8c9d6ce3546c429b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ea33b21aafd497d8c9d6ce3546c429b. Latest consensus state: current_term: 1 leader_uuid: "8ea33b21aafd497d8c9d6ce3546c429b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea33b21aafd497d8c9d6ce3546c429b" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 39041 } } }
I20260925 22:45:29.333997 24448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ea33b21aafd497d8c9d6ce3546c429b [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:29.334367 24449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ea33b21aafd497d8c9d6ce3546c429b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ea33b21aafd497d8c9d6ce3546c429b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea33b21aafd497d8c9d6ce3546c429b" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 39041 } } }
I20260925 22:45:29.334465 24449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ea33b21aafd497d8c9d6ce3546c429b [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:29.334751 24456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:29.337814 24456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:29.344209 24456 catalog_manager.cc:1347] Generated new cluster ID: a94d1468bb674207bdd041ccee616c56
I20260925 22:45:29.344308 24456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:29.355180 24456 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:29.356416 24456 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:29.361346 24456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ea33b21aafd497d8c9d6ce3546c429b: Generated new TSK 0
I20260925 22:45:29.362108 24456 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:45:29.368569 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 24369
2026-09-25T22:45:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1431 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:29 dist-test-slave-csnl krb5kdc[24468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:29 dist-test-slave-csnl krb5kdc[24468](info): set up 2 sockets
Sep 25 22:45:29 dist-test-slave-csnl krb5kdc[24468](info): commencing operation
krb5kdc: starting...
W20260925 22:45:30.399381 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:45:30 dist-test-slave-csnl krb5kdc[24468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:45:30.526450 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:45395
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:41381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:45395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:30.641794 24481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:30.642176 24481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:30.642273 24481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:30.646595 24481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:30.646750 24481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:30.646791 24481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:30.646831 24481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:30.646864 24481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:30.652777 24481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:41381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:45395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:45395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:30.654052 24481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:30.655244 24481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:30.664600 24489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:30.664670 24481 server_base.cc:1034] running on GCE node
W20260925 22:45:30.664603 24486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:30.664605 24487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:30.665511 24481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:30.666815 24481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:30.667996 24481 hybrid_clock.cc:648] HybridClock initialized: now 1790376330667985 us; error 38 us; skew 500 ppm
Sep 25 22:45:30 dist-test-slave-csnl krb5kdc[24468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376330, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:30.670848 24481 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:45:30.671909 24481 webserver.cc:460] Webserver started at http://127.2.74.254:38519/ using document root <none> and password file <none>
I20260925 22:45:30.672484 24481 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:30.672554 24481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:30.672731 24481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:30.675050 24481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "336fb45a3588423c94e37f9ae771da68"
format_stamp: "Formatted at 2026-09-25 22:45:30 on dist-test-slave-csnl"
server_key: "805f2213719483c9a84ed4e5cd9a2c35"
server_key_iv: "32663fb0cd0af5d4cdb92686153375f6"
server_key_version: "encryptionkey@0"
I20260925 22:45:30.675709 24481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "336fb45a3588423c94e37f9ae771da68"
format_stamp: "Formatted at 2026-09-25 22:45:30 on dist-test-slave-csnl"
server_key: "805f2213719483c9a84ed4e5cd9a2c35"
server_key_iv: "32663fb0cd0af5d4cdb92686153375f6"
server_key_version: "encryptionkey@0"
I20260925 22:45:30.680068 24481 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:45:30.682811 24496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:30.684312 24481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:45:30.684473 24481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "336fb45a3588423c94e37f9ae771da68"
format_stamp: "Formatted at 2026-09-25 22:45:30 on dist-test-slave-csnl"
server_key: "805f2213719483c9a84ed4e5cd9a2c35"
server_key_iv: "32663fb0cd0af5d4cdb92686153375f6"
server_key_version: "encryptionkey@0"
I20260925 22:45:30.684607 24481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:30.689661 24481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:30.693300 24481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:30.693533 24481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:30.701236 24481 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:45395
I20260925 22:45:30.701258 24548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:45395 every 8 connection(s)
I20260925 22:45:30.702250 24481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:30.703603 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 24481
I20260925 22:45:30.703747 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:45:30.704064 2347 external_mini_cluster.cc:1428] Setting key aa7508395bbea9e38264fecfe7b0061f
I20260925 22:45:30.705541 24549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:30.709465 24552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:30.705532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35045 (local address 127.2.74.254:45395)
0925 22:45:30.705997 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:30.706013 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:30.706049 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:45:30.706696 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:30.706713 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:30.707090 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:30.707471 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:30.708262 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:30.708267 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:30.708279 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:30.708289 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:30.708341 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:30.708359 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:30.708509 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:30.708593 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 22:45:30.708756 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":143,"threads_started":1}
I20260925 22:45:30.712005 24549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 336fb45a3588423c94e37f9ae771da68: Bootstrap starting.
I20260925 22:45:30.712627 2347 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:45:30.714610 24549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 336fb45a3588423c94e37f9ae771da68: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:30.715623 24549 log.cc:824] T 00000000000000000000000000000000 P 336fb45a3588423c94e37f9ae771da68: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:30.716858 24552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:30.714870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46351 (local address 127.2.74.254:45395)
0925 22:45:30.715052 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:30.715055 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:30.715278 (+ 223us) server_negotiation.cc:407] Connection header received
0925 22:45:30.715425 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:30.715427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:30.715465 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:30.715527 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:30.716410 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:30.716413 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:30.716416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:30.716429 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:30.716461 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:30.716479 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:30.716585 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:30.716681 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 22:45:30.716713 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 22:45:30.718093 24549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 336fb45a3588423c94e37f9ae771da68: No bootstrap required, opened a new log
I20260925 22:45:30.721596 24549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 336fb45a3588423c94e37f9ae771da68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "336fb45a3588423c94e37f9ae771da68" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 45395 } }
I20260925 22:45:30.721834 24549 raft_consensus.cc:374] T 00000000000000000000000000000000 P 336fb45a3588423c94e37f9ae771da68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:30.721868 24549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 336fb45a3588423c94e37f9ae771da68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 336fb45a3588423c94e37f9ae771da68, State: Initialized, Role: FOLLOWER
I20260925 22:45:30.722287 24549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 336fb45a3588423c94e37f9ae771da68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "336fb45a3588423c94e37f9ae771da68" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 45395 } }
I20260925 22:45:30.722411 24549 raft_consensus.cc:388] T 00000000000000000000000000000000 P 336fb45a3588423c94e37f9ae771da68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:30.722453 24549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 336fb45a3588423c94e37f9ae771da68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:30.722518 24549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 336fb45a3588423c94e37f9ae771da68 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:30.723644 24549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 336fb45a3588423c94e37f9ae771da68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "336fb45a3588423c94e37f9ae771da68" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 45395 } }
I20260925 22:45:30.724030 24549 leader_election.cc:302] T 00000000000000000000000000000000 P 336fb45a3588423c94e37f9ae771da68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 336fb45a3588423c94e37f9ae771da68; no voters:
I20260925 22:45:30.724404 24549 leader_election.cc:288] T 00000000000000000000000000000000 P 336fb45a3588423c94e37f9ae771da68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:30.724565 24559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 336fb45a3588423c94e37f9ae771da68 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:30.724808 24559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 336fb45a3588423c94e37f9ae771da68 [term 1 LEADER]: Becoming Leader. State: Replica: 336fb45a3588423c94e37f9ae771da68, State: Running, Role: LEADER
I20260925 22:45:30.725131 24559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 336fb45a3588423c94e37f9ae771da68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "336fb45a3588423c94e37f9ae771da68" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 45395 } }
I20260925 22:45:30.725680 24549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 336fb45a3588423c94e37f9ae771da68 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:30.727043 24560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 336fb45a3588423c94e37f9ae771da68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "336fb45a3588423c94e37f9ae771da68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "336fb45a3588423c94e37f9ae771da68" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 45395 } } }
I20260925 22:45:30.727174 24560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 336fb45a3588423c94e37f9ae771da68 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:30.727500 24561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 336fb45a3588423c94e37f9ae771da68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 336fb45a3588423c94e37f9ae771da68. Latest consensus state: current_term: 1 leader_uuid: "336fb45a3588423c94e37f9ae771da68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "336fb45a3588423c94e37f9ae771da68" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 45395 } } }
I20260925 22:45:30.727596 24561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 336fb45a3588423c94e37f9ae771da68 [sys.catalog]: This master's current role is: LEADER
W20260925 22:45:30.730314 24574 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 336fb45a3588423c94e37f9ae771da68: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:45:30.730412 24574 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:45:30.730497 24569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:30.731317 24569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:30.737618 24569 catalog_manager.cc:1347] Generated new cluster ID: bc1f609e8f6b45e6b4ed6bcccdd2e16a
I20260925 22:45:30.737712 24569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:30.747730 24569 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:30.748127 24569 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:30.757256 24569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 336fb45a3588423c94e37f9ae771da68: Generated new TSK 0
I20260925 22:45:30.758195 24569 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:45:30.763034 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 24481
2026-09-25T22:45:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1393 ms)
[----------] 18 tests from AuthTokenIssuingTest (25872 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:30 dist-test-slave-csnl krb5kdc[24580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:30 dist-test-slave-csnl krb5kdc[24580](info): set up 2 sockets
Sep 25 22:45:30 dist-test-slave-csnl krb5kdc[24580](info): commencing operation
krb5kdc: starting...
W20260925 22:45:31.803333 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:45:31 dist-test-slave-csnl krb5kdc[24580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:45:31.929922 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:40551
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:45237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:40551,127.2.74.253:42295,127.2.74.252:44629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:32.045564 24593 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:45:32.045961 24593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:32.046041 24593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:32.046118 24593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:32.050254 24593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:32.050366 24593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:32.050400 24593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:32.050432 24593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:32.050467 24593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:32.055972 24593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:45237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:40551,127.2.74.253:42295,127.2.74.252:44629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:40551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:32.057147 24593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:32.058212 24593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:45:32.067260 24593 server_base.cc:1034] running on GCE node
W20260925 22:45:32.067209 24601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:32.067203 24598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:32.067203 24599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:32.068195 24593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:32.069464 24593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:32.070679 24593 hybrid_clock.cc:648] HybridClock initialized: now 1790376332070663 us; error 36 us; skew 500 ppm
Sep 25 22:45:32 dist-test-slave-csnl krb5kdc[24580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376332, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:32.073760 24593 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:45:32.074787 24593 webserver.cc:460] Webserver started at http://127.2.74.254:35851/ using document root <none> and password file <none>
I20260925 22:45:32.075430 24593 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:32.075505 24593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:32.075692 24593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:32.077986 24593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "8cbe7d32e0984b049ca2e3a81d3b5727"
format_stamp: "Formatted at 2026-09-25 22:45:32 on dist-test-slave-csnl"
server_key: "7b939b04b3e0c3e677bd57a13b35ba78"
server_key_iv: "d6b13eb3acdb5b47c785110cbcf278ae"
server_key_version: "encryptionkey@0"
I20260925 22:45:32.079008 24593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "8cbe7d32e0984b049ca2e3a81d3b5727"
format_stamp: "Formatted at 2026-09-25 22:45:32 on dist-test-slave-csnl"
server_key: "7b939b04b3e0c3e677bd57a13b35ba78"
server_key_iv: "d6b13eb3acdb5b47c785110cbcf278ae"
server_key_version: "encryptionkey@0"
I20260925 22:45:32.083508 24593 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:45:32.086644 24608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:32.088310 24593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:32.088505 24593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "8cbe7d32e0984b049ca2e3a81d3b5727"
format_stamp: "Formatted at 2026-09-25 22:45:32 on dist-test-slave-csnl"
server_key: "7b939b04b3e0c3e677bd57a13b35ba78"
server_key_iv: "d6b13eb3acdb5b47c785110cbcf278ae"
server_key_version: "encryptionkey@0"
I20260925 22:45:32.088651 24593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:32.111047 24593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:32.114413 24593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:32.114645 24593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:32.122282 24593 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:40551
I20260925 22:45:32.122682 24660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:40551 every 8 connection(s)
I20260925 22:45:32.123373 24593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:32.126263 24661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:32.127012 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 24593
I20260925 22:45:32.127169 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:45:32.127436 2347 external_mini_cluster.cc:1428] Setting key 51b9b12e99cae9cc5d977d8b111f9052
I20260925 22:45:32.130499 24661 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40551 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.74.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.253@KRBTEST.COM" created.
Sep 25 22:45:32 dist-test-slave-csnl krb5kdc[24580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376332, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:45:32.147863 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.133218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:40551 (local address 127.0.0.1:54324)
0925 22:45:32.133877 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:32.133918 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:32.134863 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:32.135738 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:32.135743 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:32.136192 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:32.136514 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:32.136529 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.137879 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.137881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:32.139041 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:32.139057 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.139812 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.139815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:32.139930 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:32.140587 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:32.140606 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:32.142398 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:32.144609 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:32.144621 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:32.144633 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:32.144884 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:32.145236 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:32.145239 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:32.145241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:32.145359 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:32.145996 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:32.146003 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:32.146157 (+ 154us) client_negotiation.cc:769] Sending connection context
0925 22:45:32.146344 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 22:45:32.146771 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":103,"spinlock_wait_cycles":22912,"thread_start_us":186,"threads_started":1}
I20260925 22:45:32.147863 24664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.133217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54324 (local address 127.2.74.254:40551)
0925 22:45:32.133714 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:32.133735 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:32.134042 (+ 307us) server_negotiation.cc:407] Connection header received
0925 22:45:32.135267 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:32.135273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:32.135534 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:32.135740 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:32.136669 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.137703 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.139206 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.139620 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.139812 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:32.142586 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:32.142592 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:32.142604 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:32.142626 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:32.144462 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.145017 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.145023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.145024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.145136 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.145451 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.145455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.145457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.145879 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:32.146027 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:32.146406 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 22:45:32.146690 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":177,"threads_started":1}
I20260925 22:45:32.150576 24661 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.253" port: 42295 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:45:32.151609 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.151311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:42295 (local address 127.0.0.1:42992)
0925 22:45:32.151448 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:32.151525 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.253:42295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 22:45:32.151834 24611 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.74.253:42295: connect: Connection refused (error 111)
I20260925 22:45:32.152752 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.152525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:42295 (local address 127.0.0.1:42994)
0925 22:45:32.152633 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:32.152688 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.253:42295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 22:45:32.152917 24661 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.74.253:42295: Network error: Client connection negotiation failed: client connection to 127.2.74.253:42295: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.253.keytab.
Entry for principal kudu/127.2.74.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.253.keytab.
I20260925 22:45:32.154485 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.253:42295
--webserver_interface=127.2.74.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.253.keytab
--principal=kudu/127.2.74.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:45237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:40551,127.2.74.253:42295,127.2.74.252:44629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:45:32.202271 24661 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.74.253" port: 42295 } attempt: 1
I20260925 22:45:32.203274 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.202878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:42295 (local address 127.0.0.1:43004)
0925 22:45:32.203095 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:32.203175 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.253:42295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260925 22:45:32.204090 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.203839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:42295 (local address 127.0.0.1:43008)
0925 22:45:32.203961 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:32.204017 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.253:42295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 22:45:32.204245 24661 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.74.253:42295: Network error: Client connection negotiation failed: client connection to 127.2.74.253:42295: connect: Connection refused (error 111)
W20260925 22:45:32.272017 24668 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:45:32.272414 24668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:32.272478 24668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:32.272559 24668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 22:45:32.272467 24661 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.74.253" port: 42295 } attempt: 2
I20260925 22:45:32.273350 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.273003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:42295 (local address 127.0.0.1:43016)
0925 22:45:32.273184 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:32.273258 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.253:42295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 22:45:32.274114 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.273868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:42295 (local address 127.0.0.1:43018)
0925 22:45:32.273989 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:32.274042 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.253:42295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 22:45:32.274250 24661 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.74.253:42295: Network error: Client connection negotiation failed: client connection to 127.2.74.253:42295: connect: Connection refused (error 111)
W20260925 22:45:32.277527 24668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:32.277599 24668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:32.277619 24668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:32.277637 24668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:32.277652 24668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:32.283568 24668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:45237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:40551,127.2.74.253:42295,127.2.74.252:44629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.253.keytab
--principal=kudu/127.2.74.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.253:42295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.74.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:32.284704 24668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:32.285741 24668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:32.294968 24673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:32.295009 24668 server_base.cc:1034] running on GCE node
W20260925 22:45:32.294961 24676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:32.295018 24674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:32.295905 24668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:32.297175 24668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:32.298415 24668 hybrid_clock.cc:648] HybridClock initialized: now 1790376332298394 us; error 38 us; skew 500 ppm
Sep 25 22:45:32 dist-test-slave-csnl krb5kdc[24580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376332, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:32.301512 24668 init.cc:377] Logged in from keytab as kudu/127.2.74.253@KRBTEST.COM (short username kudu)
I20260925 22:45:32.302484 24668 webserver.cc:460] Webserver started at http://127.2.74.253:41973/ using document root <none> and password file <none>
I20260925 22:45:32.303122 24668 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:32.303217 24668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:32.303454 24668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:32.305583 24668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-1/data/instance:
uuid: "79f1f787f8c541e099b776308ba586b8"
format_stamp: "Formatted at 2026-09-25 22:45:32 on dist-test-slave-csnl"
server_key: "ac49509bd27a559d299a755fb1161ec7"
server_key_iv: "742ea2e7bb59ffdf844ef921671e611e"
server_key_version: "encryptionkey@0"
I20260925 22:45:32.306156 24668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-1/wal/instance:
uuid: "79f1f787f8c541e099b776308ba586b8"
format_stamp: "Formatted at 2026-09-25 22:45:32 on dist-test-slave-csnl"
server_key: "ac49509bd27a559d299a755fb1161ec7"
server_key_iv: "742ea2e7bb59ffdf844ef921671e611e"
server_key_version: "encryptionkey@0"
I20260925 22:45:32.310304 24668 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:45:32.313009 24683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:32.314325 24668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:45:32.314489 24668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-1/wal
uuid: "79f1f787f8c541e099b776308ba586b8"
format_stamp: "Formatted at 2026-09-25 22:45:32 on dist-test-slave-csnl"
server_key: "ac49509bd27a559d299a755fb1161ec7"
server_key_iv: "742ea2e7bb59ffdf844ef921671e611e"
server_key_version: "encryptionkey@0"
I20260925 22:45:32.314615 24668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:32.328893 24668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:32.332506 24668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:32.332736 24668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:32.340776 24668 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.253:42295
I20260925 22:45:32.341094 24735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.253:42295 every 8 connection(s)
I20260925 22:45:32.341884 24668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-1/data/info.pb
I20260925 22:45:32.344887 24736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:32.348871 24736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40551 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:45:32.351601 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 24668
I20260925 22:45:32.351715 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-1/wal/instance
I20260925 22:45:32.352053 2347 external_mini_cluster.cc:1428] Setting key 86637ab1f8507fb703b05f759b3c34ed
WARNING: no policy specified for kudu/127.2.74.252@KRBTEST.COM; defaulting to no policy
Sep 25 22:45:32 dist-test-slave-csnl krb5kdc[24580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376332, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.253@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.252@KRBTEST.COM" created.
I20260925 22:45:32.365398 24661 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.74.253" port: 42295 } attempt: 3
I20260925 22:45:32.365600 24664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.351023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54332 (local address 127.2.74.254:40551)
0925 22:45:32.351199 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:32.351204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:32.352302 (+ 1098us) server_negotiation.cc:407] Connection header received
0925 22:45:32.353932 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:32.353939 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:32.354015 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:32.354111 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:32.355501 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.356415 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.357723 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.358178 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.358305 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:32.361267 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:32.361271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:32.361273 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:32.361329 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:32.362886 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.363684 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.363689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.363691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.363794 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.364167 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.364173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.364178 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.364462 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:32.364572 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:32.365205 (+ 633us) server_negotiation.cc:299] Negotiation successful
0925 22:45:32.365350 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 22:45:32.366627 24738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.351461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:40551 (local address 127.0.0.1:54332)
0925 22:45:32.352091 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:32.352164 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:32.353588 (+ 1424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:32.354292 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:32.354299 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:32.354667 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:32.355325 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:32.355341 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.356567 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.356570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:32.357566 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:32.357586 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.358376 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.358380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:32.358548 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:32.359316 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:32.359335 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:32.361047 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:32.363125 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:32.363135 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:32.363155 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:32.363522 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:32.363901 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:32.363903 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:32.363905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:32.364057 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:32.364596 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:32.364601 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:32.364918 (+ 317us) client_negotiation.cc:769] Sending connection context
0925 22:45:32.365129 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 22:45:32.365506 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":160,"threads_started":1}
I20260925 22:45:32.368676 24736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.253" port: 42295 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:45:32 dist-test-slave-csnl krb5kdc[24580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376332, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for kudu/127.2.74.253@KRBTEST.COM
Sep 25 22:45:32 dist-test-slave-csnl krb5kdc[24580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376332, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.253@KRBTEST.COM for kudu/127.2.74.253@KRBTEST.COM
I20260925 22:45:32.376963 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.365888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:42295 (local address 127.0.0.1:43032)
0925 22:45:32.366281 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:32.366297 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:32.366377 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:32.367137 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:32.367140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:32.367329 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:32.367459 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:32.367466 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.368620 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.368623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:32.369653 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:32.369669 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.370343 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.370347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:32.370470 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:32.371098 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:32.371114 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:32.372742 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:32.375009 (+ 2267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:32.375013 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:32.375015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:32.375332 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:32.375684 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:32.375687 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:32.375690 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:32.375794 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:32.376316 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:32.376320 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:32.376443 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 22:45:32.376557 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 22:45:32.376723 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310}
I20260925 22:45:32.377481 24741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.366119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43032 (local address 127.2.74.253:42295)
0925 22:45:32.366630 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:32.366644 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:32.366670 (+ 26us) server_negotiation.cc:407] Connection header received
0925 22:45:32.366719 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:32.366727 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:32.366943 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:32.367130 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:32.367586 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.368502 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.369781 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.370180 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.370719 (+ 539us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:32.372915 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:32.372931 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:32.372951 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:32.372984 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:32.374823 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.375469 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.375478 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.375483 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.375593 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.375915 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.375918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.375919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.376189 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:32.376341 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:32.377169 (+ 828us) server_negotiation.cc:299] Negotiation successful
0925 22:45:32.377332 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":396,"thread_start_us":176,"threads_started":1}
I20260925 22:45:32.379694 24661 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.252" port: 44629 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:45:32.380620 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.380291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:44629 (local address 127.0.0.1:58084)
0925 22:45:32.380468 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:32.380529 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:44629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 22:45:32.380770 24738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.369240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:42295 (local address 127.0.0.1:43046)
0925 22:45:32.369356 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:32.369370 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:32.369459 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:32.370272 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:32.370278 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:32.370487 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:32.370608 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:32.370619 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.371757 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.371761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:32.372269 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:32.372275 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.372791 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.372794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:32.372904 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:32.373421 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:32.373433 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:32.375887 (+ 2454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:32.378935 (+ 3048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:32.378939 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:32.378941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:32.379215 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:32.379589 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:32.379592 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:32.379596 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:32.379685 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:32.380163 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:32.380169 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:32.380275 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 22:45:32.380384 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 22:45:32.380565 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":4}
I20260925 22:45:32.380817 24742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.369396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43046 (local address 127.2.74.253:42295)
0925 22:45:32.369998 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:32.370009 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:32.370043 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:45:32.370085 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:32.370090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:32.370141 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:32.370235 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:32.370728 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.371632 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.372440 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.372678 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.372789 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:32.376035 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:32.376038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:32.376040 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:32.376060 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:32.378786 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.379326 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.379330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.379331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.379457 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.379786 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.379789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.379791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.380052 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:32.380151 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:32.380450 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 22:45:32.380635 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":512,"thread_start_us":120,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.252.keytab.
Entry for principal kudu/127.2.74.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.252.keytab.
I20260925 22:45:32.381521 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.381195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:44629 (local address 127.0.0.1:58098)
0925 22:45:32.381372 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:32.381435 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:44629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260925 22:45:32.381511 24736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.252" port: 44629 } has no permanent_uuid. Determining permanent_uuid...
W20260925 22:45:32.381724 24661 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.74.252:44629: Network error: Client connection negotiation failed: client connection to 127.2.74.252:44629: connect: Connection refused (error 111)
I20260925 22:45:32.382206 24738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.381906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:44629 (local address 127.0.0.1:58110)
0925 22:45:32.382034 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:32.382127 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:44629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 22:45:32.382405 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.252:44629
--webserver_interface=127.2.74.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.252.keytab
--principal=kudu/127.2.74.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:45237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:40551,127.2.74.253:42295,127.2.74.252:44629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:32.382443 24685 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.74.252:44629: connect: Connection refused (error 111)
I20260925 22:45:32.383443 24738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.383168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:44629 (local address 127.0.0.1:58116)
0925 22:45:32.383302 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:32.383363 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:44629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 22:45:32.383594 24736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.74.252:44629: Network error: Client connection negotiation failed: client connection to 127.2.74.252:44629: connect: Connection refused (error 111)
I20260925 22:45:32.412918 24661 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.74.252" port: 44629 } attempt: 1
I20260925 22:45:32.413856 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.413514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:44629 (local address 127.0.0.1:58128)
0925 22:45:32.413697 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:32.413764 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:44629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 22:45:32.414706 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.414467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:44629 (local address 127.0.0.1:58138)
0925 22:45:32.414602 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:32.414651 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:44629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 22:45:32.415000 24661 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.74.252:44629: Network error: Client connection negotiation failed: client connection to 127.2.74.252:44629: connect: Connection refused (error 111)
I20260925 22:45:32.432911 24736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.74.252" port: 44629 } attempt: 1
I20260925 22:45:32.433907 24738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.433551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:44629 (local address 127.0.0.1:58150)
0925 22:45:32.433748 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:32.433826 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:44629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260925 22:45:32.435109 24738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.434544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:44629 (local address 127.0.0.1:58164)
0925 22:45:32.434793 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:32.434892 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:44629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260925 22:45:32.435338 24736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.74.252:44629: Network error: Client connection negotiation failed: client connection to 127.2.74.252:44629: connect: Connection refused (error 111)
I20260925 22:45:32.490221 24661 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.74.252" port: 44629 } attempt: 2
I20260925 22:45:32.491256 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.490877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:44629 (local address 127.0.0.1:58172)
0925 22:45:32.491080 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:32.491159 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:44629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260925 22:45:32.492028 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.491804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:44629 (local address 127.0.0.1:58188)
0925 22:45:32.491924 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:32.491971 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:44629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 22:45:32.492355 24661 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.74.252:44629: Network error: Client connection negotiation failed: client connection to 127.2.74.252:44629: connect: Connection refused (error 111)
I20260925 22:45:32.503558 24736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.74.252" port: 44629 } attempt: 2
W20260925 22:45:32.503808 24744 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:45:32.504242 24744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:32.504292 24744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:32.504364 24744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 22:45:32.504563 24738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.504165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:44629 (local address 127.0.0.1:58204)
0925 22:45:32.504383 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:32.504479 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:44629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 22:45:32.505304 24738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.505099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:44629 (local address 127.0.0.1:58220)
0925 22:45:32.505211 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:32.505255 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:44629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 22:45:32.505524 24736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.74.252:44629: Network error: Client connection negotiation failed: client connection to 127.2.74.252:44629: connect: Connection refused (error 111)
W20260925 22:45:32.509243 24744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:32.509306 24744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:32.509326 24744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:32.509344 24744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:32.509387 24744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:32.515280 24744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:45237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:40551,127.2.74.253:42295,127.2.74.252:44629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.252.keytab
--principal=kudu/127.2.74.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.252:44629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.74.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:32.516492 24744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:32.517652 24744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:32.526804 24752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:32.526801 24750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:32.526778 24749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:32.527238 24744 server_base.cc:1034] running on GCE node
I20260925 22:45:32.527881 24744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:32.529186 24744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:32.530368 24744 hybrid_clock.cc:648] HybridClock initialized: now 1790376332530332 us; error 54 us; skew 500 ppm
Sep 25 22:45:32 dist-test-slave-csnl krb5kdc[24580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376332, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:32.533547 24744 init.cc:377] Logged in from keytab as kudu/127.2.74.252@KRBTEST.COM (short username kudu)
I20260925 22:45:32.534685 24744 webserver.cc:460] Webserver started at http://127.2.74.252:36667/ using document root <none> and password file <none>
I20260925 22:45:32.535394 24744 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:32.535499 24744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:32.535758 24744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:32.538091 24744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-2/data/instance:
uuid: "4406ed3b43f1445eb16dc4646496dc79"
format_stamp: "Formatted at 2026-09-25 22:45:32 on dist-test-slave-csnl"
server_key: "4fee0d2362728b6947a677ab00fc5f4c"
server_key_iv: "885b37066e3cdc354050b6aadaf961dc"
server_key_version: "encryptionkey@0"
I20260925 22:45:32.538789 24744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-2/wal/instance:
uuid: "4406ed3b43f1445eb16dc4646496dc79"
format_stamp: "Formatted at 2026-09-25 22:45:32 on dist-test-slave-csnl"
server_key: "4fee0d2362728b6947a677ab00fc5f4c"
server_key_iv: "885b37066e3cdc354050b6aadaf961dc"
server_key_version: "encryptionkey@0"
I20260925 22:45:32.543229 24744 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:45:32.546114 24759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:32.547669 24744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:32.547868 24744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-2/wal
uuid: "4406ed3b43f1445eb16dc4646496dc79"
format_stamp: "Formatted at 2026-09-25 22:45:32 on dist-test-slave-csnl"
server_key: "4fee0d2362728b6947a677ab00fc5f4c"
server_key_iv: "885b37066e3cdc354050b6aadaf961dc"
server_key_version: "encryptionkey@0"
I20260925 22:45:32.548013 24744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:32.566931 24744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:32.571121 24744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:32.571373 24744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:32.578998 24744 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.252:44629
I20260925 22:45:32.579331 24811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.252:44629 every 8 connection(s)
I20260925 22:45:32.580137 24744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-2/data/info.pb
I20260925 22:45:32.583359 24812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:32.587448 24812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40551 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:45:32.589816 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 24744
I20260925 22:45:32.589956 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376047580784-2347-0/minicluster-data/master-2/wal/instance
I20260925 22:45:32.590240 2347 external_mini_cluster.cc:1428] Setting key 65c427094858a1436d8c5d812ad67566
I20260925 22:45:32.591532 24661 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.74.252" port: 44629 } attempt: 3
I20260925 22:45:32.596704 24736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.74.252" port: 44629 } attempt: 3
Sep 25 22:45:32 dist-test-slave-csnl krb5kdc[24580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376332, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for kudu/127.2.74.252@KRBTEST.COM
Sep 25 22:45:32 dist-test-slave-csnl krb5kdc[24580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376332, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.252@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
Sep 25 22:45:32 dist-test-slave-csnl krb5kdc[24580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.252@KRBTEST.COM
I20260925 22:45:32.604820 24664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.589536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54344 (local address 127.2.74.254:40551)
0925 22:45:32.589711 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:32.589716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:32.590731 (+ 1015us) server_negotiation.cc:407] Connection header received
0925 22:45:32.592102 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:32.592105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:32.592159 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:32.592211 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:32.593542 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.594153 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.595342 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.595732 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.595828 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:32.600141 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:32.600145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:32.600148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:32.600184 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:32.602200 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.602863 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.602868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.602870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.602981 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.603391 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.603394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.603396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.603716 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:32.603804 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:32.604493 (+ 689us) server_negotiation.cc:299] Negotiation successful
0925 22:45:32.604638 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":65,"spinlock_wait_cycles":2176}
Sep 25 22:45:32 dist-test-slave-csnl krb5kdc[24580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376332, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.253@KRBTEST.COM for kudu/127.2.74.252@KRBTEST.COM
I20260925 22:45:32.605234 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.592042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:44629 (local address 127.0.0.1:58236)
0925 22:45:32.592222 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:32.592235 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:32.592289 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:32.592657 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:32.592661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:32.592882 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:32.593002 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:32.593008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.594856 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.594859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:32.595641 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:32.595651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.596709 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.596714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:32.596878 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:32.597472 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:32.597487 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:32.599134 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:32.602735 (+ 3601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:32.602738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:32.602740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:32.603057 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:32.603391 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:32.603394 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:32.603396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:32.603892 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:32.604342 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:32.604345 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:32.604454 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 22:45:32.604548 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 22:45:32.605097 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":481,"spinlock_wait_cycles":3456}
I20260925 22:45:32.606199 24816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.590975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58234 (local address 127.2.74.252:44629)
0925 22:45:32.591407 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:32.591418 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:32.591463 (+ 45us) server_negotiation.cc:407] Connection header received
0925 22:45:32.591868 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:32.591897 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:32.592191 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:32.592431 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:32.593391 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.594735 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.596122 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.596542 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.597035 (+ 493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:32.600605 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:32.600612 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:32.600615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:32.600643 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:32.602672 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.603215 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.603219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.603221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.603303 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.603626 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.603629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.603631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.604151 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:32.604364 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:32.605063 (+ 699us) server_negotiation.cc:299] Negotiation successful
0925 22:45:32.605190 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":495,"server-negotiator.queue_time_us":318,"thread_start_us":169,"threads_started":1}
I20260925 22:45:32.606200 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.592083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58236 (local address 127.2.74.252:44629)
0925 22:45:32.592408 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:32.592416 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:32.592428 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:45:32.592460 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:32.592462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:32.592497 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:32.593094 (+ 597us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:32.593176 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.594735 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.595759 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.596464 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.596582 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:32.599296 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:32.599302 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:32.599314 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:32.599335 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:32.602605 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.603180 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.603186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.603187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.603282 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.603993 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.603996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.603998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.604224 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:32.604647 (+ 423us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:32.604772 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 22:45:32.605095 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":325,"server-negotiator.queue_time_us":265,"spinlock_wait_cycles":16512,"thread_start_us":120,"threads_started":1}
I20260925 22:45:32.606200 24814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.589924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:40551 (local address 127.0.0.1:54344)
0925 22:45:32.590534 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:32.590599 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:32.591856 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:32.592466 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:32.592473 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:32.593046 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:32.593399 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:32.593420 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.594260 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.594265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:32.595231 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:32.595241 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.595994 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.595998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:32.596264 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:32.597500 (+ 1236us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:32.597523 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:32.599948 (+ 2425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:32.602344 (+ 2396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:32.602353 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:32.602365 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:32.602756 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:32.603102 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:32.603105 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:32.603109 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:32.603293 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:32.604037 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:32.604047 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:32.604234 (+ 187us) client_negotiation.cc:769] Sending connection context
0925 22:45:32.605448 (+ 1214us) client_negotiation.cc:240] Negotiation successful
0925 22:45:32.605637 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":27,"spinlock_wait_cycles":6912,"thread_start_us":129,"threads_started":1}
I20260925 22:45:32.609762 24738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.597186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:44629 (local address 127.0.0.1:58244)
0925 22:45:32.597314 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:32.597329 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:32.597414 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:32.598547 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:32.598550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:32.598692 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:32.598778 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:32.598783 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.600413 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.600416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:32.602480 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:32.602498 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.603156 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.603163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:32.603266 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:32.603806 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:32.603821 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:32.605826 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:32.608070 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:32.608074 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:32.608076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:32.608357 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:32.608663 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:32.608666 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:32.608668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:32.608755 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:32.609214 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:32.609219 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:32.609321 (+ 102us) client_negotiation.cc:769] Sending connection context
0925 22:45:32.609439 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 22:45:32.609587 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 22:45:32.609777 24818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.597230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58244 (local address 127.2.74.252:44629)
0925 22:45:32.598284 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:32.598292 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:32.598305 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:45:32.598353 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:32.598356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:32.598404 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:32.598510 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:32.598936 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.600259 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.602650 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.603015 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.603650 (+ 635us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:32.605981 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:32.605984 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:32.605987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:32.606010 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:32.607947 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.608483 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.608485 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.608486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.608554 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.608896 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.608899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.608901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.609124 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:32.609211 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:32.609500 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 22:45:32.609612 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":985,"thread_start_us":121,"threads_started":1}
I20260925 22:45:32.611219 24812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.253" port: 42295 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:45:32.611672 24661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cbe7d32e0984b049ca2e3a81d3b5727: Bootstrap starting.
I20260925 22:45:32.613070 24736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79f1f787f8c541e099b776308ba586b8: Bootstrap starting.
I20260925 22:45:32.614727 24661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cbe7d32e0984b049ca2e3a81d3b5727: Neither blocks nor log segments found. Creating new log.
Sep 25 22:45:32 dist-test-slave-csnl krb5kdc[24580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:45:32.615567 24736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79f1f787f8c541e099b776308ba586b8: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:32.615737 24661 log.cc:824] T 00000000000000000000000000000000 P 8cbe7d32e0984b049ca2e3a81d3b5727: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:32.616504 24736 log.cc:824] T 00000000000000000000000000000000 P 79f1f787f8c541e099b776308ba586b8: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:45:32 dist-test-slave-csnl krb5kdc[24580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376332, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.252@KRBTEST.COM for kudu/127.2.74.253@KRBTEST.COM
I20260925 22:45:32.619284 24736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79f1f787f8c541e099b776308ba586b8: No bootstrap required, opened a new log
I20260925 22:45:32.619310 24661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cbe7d32e0984b049ca2e3a81d3b5727: No bootstrap required, opened a new log
I20260925 22:45:32.620710 24664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.608942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54350 (local address 127.2.74.254:40551)
0925 22:45:32.609059 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:32.609063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:32.609119 (+ 56us) server_negotiation.cc:407] Connection header received
0925 22:45:32.609269 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:32.609272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:32.609319 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:32.609381 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:32.610309 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.611046 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.612433 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.612731 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.612807 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:32.616197 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:32.616201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:32.616204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:32.616229 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:32.618427 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.619115 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.619120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.619121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.619220 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.619554 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.619558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.619559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.619807 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:32.620001 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:32.620396 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 22:45:32.620544 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:45:32.621169 24742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.611783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43054 (local address 127.2.74.253:42295)
0925 22:45:32.611898 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:32.611903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:32.611920 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:45:32.612065 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:32.612070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:32.612125 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:32.612180 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:32.612728 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.613425 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.614480 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.614815 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.614905 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:32.617319 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:32.617323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:32.617327 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:32.617361 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:32.619113 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.619695 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.619698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.619700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.619794 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.620140 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.620144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.620145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.620414 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:32.620503 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:32.620876 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 22:45:32.621003 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:45:32.621631 24814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.611688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:42295 (local address 127.0.0.1:43054)
0925 22:45:32.611868 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:32.611883 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:32.611956 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:32.612312 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:32.612315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:32.612485 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:32.612602 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:32.612609 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.613560 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.613563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:32.614346 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:32.614357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.615085 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.615089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:32.615190 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:32.615660 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:32.615674 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:32.617157 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:32.619285 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:32.619294 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:32.619298 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:32.619582 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:32.619930 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:32.619933 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:32.619934 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:32.620030 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:32.620581 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:32.620584 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:32.620683 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 22:45:32.621294 (+ 611us) client_negotiation.cc:240] Negotiation successful
0925 22:45:32.621457 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260925 22:45:32.622577 24812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.252" port: 44629 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:45:32.623646 24736 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79f1f787f8c541e099b776308ba586b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cbe7d32e0984b049ca2e3a81d3b5727" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40551 } } peers { permanent_uuid: "79f1f787f8c541e099b776308ba586b8" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 42295 } } peers { permanent_uuid: "4406ed3b43f1445eb16dc4646496dc79" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 44629 } }
I20260925 22:45:32.623646 24661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cbe7d32e0984b049ca2e3a81d3b5727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cbe7d32e0984b049ca2e3a81d3b5727" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40551 } } peers { permanent_uuid: "79f1f787f8c541e099b776308ba586b8" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 42295 } } peers { permanent_uuid: "4406ed3b43f1445eb16dc4646496dc79" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 44629 } }
I20260925 22:45:32.623903 24661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cbe7d32e0984b049ca2e3a81d3b5727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cbe7d32e0984b049ca2e3a81d3b5727, State: Initialized, Role: FOLLOWER
I20260925 22:45:32.623893 24736 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79f1f787f8c541e099b776308ba586b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79f1f787f8c541e099b776308ba586b8, State: Initialized, Role: FOLLOWER
I20260925 22:45:32.624267 24661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cbe7d32e0984b049ca2e3a81d3b5727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cbe7d32e0984b049ca2e3a81d3b5727" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40551 } } peers { permanent_uuid: "79f1f787f8c541e099b776308ba586b8" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 42295 } } peers { permanent_uuid: "4406ed3b43f1445eb16dc4646496dc79" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 44629 } }
I20260925 22:45:32.624267 24736 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79f1f787f8c541e099b776308ba586b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cbe7d32e0984b049ca2e3a81d3b5727" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40551 } } peers { permanent_uuid: "79f1f787f8c541e099b776308ba586b8" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 42295 } } peers { permanent_uuid: "4406ed3b43f1445eb16dc4646496dc79" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 44629 } }
I20260925 22:45:32.625808 24661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cbe7d32e0984b049ca2e3a81d3b5727 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:32.625808 24736 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79f1f787f8c541e099b776308ba586b8 [sys.catalog]: configured and running, proceeding with master startup.
Sep 25 22:45:32 dist-test-slave-csnl krb5kdc[24580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376332, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.252@KRBTEST.COM for kudu/127.2.74.252@KRBTEST.COM
Sep 25 22:45:32 dist-test-slave-csnl krb5kdc[24580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.253@KRBTEST.COM
I20260925 22:45:32.631186 24822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79f1f787f8c541e099b776308ba586b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cbe7d32e0984b049ca2e3a81d3b5727" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40551 } } peers { permanent_uuid: "79f1f787f8c541e099b776308ba586b8" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 42295 } } peers { permanent_uuid: "4406ed3b43f1445eb16dc4646496dc79" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 44629 } } }
I20260925 22:45:32.631348 24822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79f1f787f8c541e099b776308ba586b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:45:32.631636 24821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cbe7d32e0984b049ca2e3a81d3b5727 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cbe7d32e0984b049ca2e3a81d3b5727" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40551 } } peers { permanent_uuid: "79f1f787f8c541e099b776308ba586b8" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 42295 } } peers { permanent_uuid: "4406ed3b43f1445eb16dc4646496dc79" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 44629 } } }
I20260925 22:45:32.631762 24821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cbe7d32e0984b049ca2e3a81d3b5727 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 22:45:32.633687 24833 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8cbe7d32e0984b049ca2e3a81d3b5727: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:45:32.633776 24833 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:45:32.634490 24742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.622615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43058 (local address 127.2.74.253:42295)
0925 22:45:32.622717 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:32.622721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:32.623120 (+ 399us) server_negotiation.cc:407] Connection header received
0925 22:45:32.623328 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:32.623332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:32.623381 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:32.623433 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:32.624378 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.625179 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.626272 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.626616 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.626702 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:32.630352 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:32.630356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:32.630359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:32.630385 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:32.632400 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.632952 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.632957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.632962 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.633058 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.633374 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.633377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.633379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.633626 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:32.633855 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:32.634198 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 22:45:32.634325 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 22:45:32.635569 24814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.623338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:44629 (local address 127.0.0.1:58248)
0925 22:45:32.623573 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:32.623585 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:32.623655 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:32.623978 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:32.623982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:32.624160 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:32.624245 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:32.624252 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.625298 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.625301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:32.626128 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:32.626137 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.626831 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.626835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:32.626910 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:32.627716 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:32.627734 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:32.630139 (+ 2405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:32.633955 (+ 3816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:32.633958 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:32.633960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:32.634206 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:32.634519 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:32.634522 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:32.634525 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:32.634617 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:32.635064 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:32.635067 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:32.635146 (+ 79us) client_negotiation.cc:769] Sending connection context
0925 22:45:32.635230 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 22:45:32.635391 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187}
I20260925 22:45:32.636204 24818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.623229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58248 (local address 127.2.74.252:44629)
0925 22:45:32.623504 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:32.623510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:32.623730 (+ 220us) server_negotiation.cc:407] Connection header received
0925 22:45:32.623757 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:32.623760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:32.623802 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:32.623868 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:32.624367 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.625173 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.626262 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.626628 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.626713 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:32.630333 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:32.630337 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:32.630339 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:32.630365 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:32.633828 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.634320 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.634323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.634340 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.634425 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.634713 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.634716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.634718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.634969 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:32.635859 (+ 890us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:32.635935 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 22:45:32.636061 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
W20260925 22:45:32.639650 24844 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79f1f787f8c541e099b776308ba586b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:45:32.639760 24844 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:45:32.640635 24812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4406ed3b43f1445eb16dc4646496dc79: Bootstrap starting.
I20260925 22:45:32.648651 24812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4406ed3b43f1445eb16dc4646496dc79: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:32.649711 24812 log.cc:824] T 00000000000000000000000000000000 P 4406ed3b43f1445eb16dc4646496dc79: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:32.643380 24593 master_runner.cc:186] Error getting master registration for 127.2.74.252:44629: OK, instance_id { permanent_uuid: "4406ed3b43f1445eb16dc4646496dc79" instance_seqno: 1790376332576901 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:45:32.651497 24593 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:45:32.648888 24668 master_runner.cc:186] Error getting master registration for 127.2.74.252:44629: OK, instance_id { permanent_uuid: "4406ed3b43f1445eb16dc4646496dc79" instance_seqno: 1790376332576901 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:45:32.652803 24668 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:45:32.653232 2347 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:45:32.653847 24812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4406ed3b43f1445eb16dc4646496dc79: No bootstrap required, opened a new log
I20260925 22:45:32.654781 24635 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8cbe7d32e0984b049ca2e3a81d3b5727"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54350
I20260925 22:45:32.655108 24635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cbe7d32e0984b049ca2e3a81d3b5727 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 22:45:32.655237 24635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cbe7d32e0984b049ca2e3a81d3b5727 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:32.657473 24635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cbe7d32e0984b049ca2e3a81d3b5727 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cbe7d32e0984b049ca2e3a81d3b5727" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40551 } } peers { permanent_uuid: "79f1f787f8c541e099b776308ba586b8" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 42295 } } peers { permanent_uuid: "4406ed3b43f1445eb16dc4646496dc79" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 44629 } }
I20260925 22:45:32.659171 24812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4406ed3b43f1445eb16dc4646496dc79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cbe7d32e0984b049ca2e3a81d3b5727" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40551 } } peers { permanent_uuid: "79f1f787f8c541e099b776308ba586b8" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 42295 } } peers { permanent_uuid: "4406ed3b43f1445eb16dc4646496dc79" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 44629 } }
I20260925 22:45:32.659401 24710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8cbe7d32e0984b049ca2e3a81d3b5727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "79f1f787f8c541e099b776308ba586b8"
I20260925 22:45:32.659512 24812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4406ed3b43f1445eb16dc4646496dc79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4406ed3b43f1445eb16dc4646496dc79, State: Initialized, Role: FOLLOWER
I20260925 22:45:32.659678 24710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79f1f787f8c541e099b776308ba586b8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:32.659897 24812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4406ed3b43f1445eb16dc4646496dc79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cbe7d32e0984b049ca2e3a81d3b5727" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40551 } } peers { permanent_uuid: "79f1f787f8c541e099b776308ba586b8" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 42295 } } peers { permanent_uuid: "4406ed3b43f1445eb16dc4646496dc79" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 44629 } }
I20260925 22:45:32.661160 24710 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 79f1f787f8c541e099b776308ba586b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cbe7d32e0984b049ca2e3a81d3b5727 in term 1.
I20260925 22:45:32.661535 24812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4406ed3b43f1445eb16dc4646496dc79 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:32.661755 24611 leader_election.cc:302] T 00000000000000000000000000000000 P 8cbe7d32e0984b049ca2e3a81d3b5727 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79f1f787f8c541e099b776308ba586b8, 8cbe7d32e0984b049ca2e3a81d3b5727; no voters:
I20260925 22:45:32.662168 24821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cbe7d32e0984b049ca2e3a81d3b5727 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:32.662403 24821 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cbe7d32e0984b049ca2e3a81d3b5727 [term 1 LEADER]: Becoming Leader. State: Replica: 8cbe7d32e0984b049ca2e3a81d3b5727, State: Running, Role: LEADER
I20260925 22:45:32.662786 24821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cbe7d32e0984b049ca2e3a81d3b5727 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cbe7d32e0984b049ca2e3a81d3b5727" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40551 } } peers { permanent_uuid: "79f1f787f8c541e099b776308ba586b8" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 42295 } } peers { permanent_uuid: "4406ed3b43f1445eb16dc4646496dc79" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 44629 } }
I20260925 22:45:32.664412 24635 leader_election.cc:288] T 00000000000000000000000000000000 P 8cbe7d32e0984b049ca2e3a81d3b5727 [CANDIDATE]: Term 1 election: Requested vote from peers 79f1f787f8c541e099b776308ba586b8 (127.2.74.253:42295), 4406ed3b43f1445eb16dc4646496dc79 (127.2.74.252:44629)
I20260925 22:45:32.664888 24821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cbe7d32e0984b049ca2e3a81d3b5727 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cbe7d32e0984b049ca2e3a81d3b5727. Latest consensus state: current_term: 1 leader_uuid: "8cbe7d32e0984b049ca2e3a81d3b5727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cbe7d32e0984b049ca2e3a81d3b5727" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40551 } } peers { permanent_uuid: "79f1f787f8c541e099b776308ba586b8" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 42295 } } peers { permanent_uuid: "4406ed3b43f1445eb16dc4646496dc79" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 44629 } } }
I20260925 22:45:32.665021 24821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cbe7d32e0984b049ca2e3a81d3b5727 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:32.665400 24848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4406ed3b43f1445eb16dc4646496dc79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cbe7d32e0984b049ca2e3a81d3b5727" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40551 } } peers { permanent_uuid: "79f1f787f8c541e099b776308ba586b8" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 42295 } } peers { permanent_uuid: "4406ed3b43f1445eb16dc4646496dc79" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 44629 } } }
I20260925 22:45:32.665546 24848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4406ed3b43f1445eb16dc4646496dc79 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:45:32.666337 24786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8cbe7d32e0984b049ca2e3a81d3b5727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4406ed3b43f1445eb16dc4646496dc79"
I20260925 22:45:32.666586 24786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4406ed3b43f1445eb16dc4646496dc79 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:32.668102 24786 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4406ed3b43f1445eb16dc4646496dc79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cbe7d32e0984b049ca2e3a81d3b5727 in term 1.
W20260925 22:45:32.668875 24860 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4406ed3b43f1445eb16dc4646496dc79: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:45:32.668965 24860 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:45:32.679296 24862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:32.680362 24862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:32.694574 24664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.673430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54366 (local address 127.2.74.254:40551)
0925 22:45:32.673563 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:32.673568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:32.675545 (+ 1977us) server_negotiation.cc:407] Connection header received
0925 22:45:32.675747 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:32.675752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:32.675804 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:32.675872 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:32.678404 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.679265 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.682453 (+ 3188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.682782 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.682878 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:32.687148 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:32.687153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:32.687155 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:32.687193 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:32.688940 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.691950 (+ 3010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.691954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.691956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.692080 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.693076 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.693080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.693081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.693368 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:32.693508 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:32.694223 (+ 715us) server_negotiation.cc:299] Negotiation successful
0925 22:45:32.694394 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:45:32.696950 24742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.674175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43062 (local address 127.2.74.253:42295)
0925 22:45:32.674291 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:32.674296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:32.677178 (+ 2882us) server_negotiation.cc:407] Connection header received
0925 22:45:32.677340 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:32.677344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:32.677395 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:32.677448 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:32.680497 (+ 3049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.681286 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.683662 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.684027 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.684108 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:32.689551 (+ 5443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:32.689555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:32.689559 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:32.689595 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:32.691396 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.692486 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.692490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.692492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.692609 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.693209 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.693213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.693215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.693454 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:32.693555 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:32.696516 (+ 2961us) server_negotiation.cc:299] Negotiation successful
0925 22:45:32.696669 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:45:32.698513 24818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:32.674879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58258 (local address 127.2.74.252:44629)
0925 22:45:32.675012 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:32.675017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:32.680437 (+ 5420us) server_negotiation.cc:407] Connection header received
0925 22:45:32.680610 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:32.680614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:32.680664 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:32.680716 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:32.681666 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.682426 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.691768 (+ 9342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:32.692153 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:32.692243 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:32.694496 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:32.694500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:32.694503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:32.694533 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:32.696296 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.696859 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.696863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.696865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.696963 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:32.697303 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:32.697309 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:32.697313 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:32.697581 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:32.697702 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:32.698080 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 22:45:32.698348 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:45:32.703320 24710 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 79f1f787f8c541e099b776308ba586b8 [term 1 FOLLOWER]: Refusing update from remote peer 8cbe7d32e0984b049ca2e3a81d3b5727: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:32.704154 24849 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8cbe7d32e0984b049ca2e3a81d3b5727 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79f1f787f8c541e099b776308ba586b8" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 42295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:32.705739 24786 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4406ed3b43f1445eb16dc4646496dc79 [term 1 FOLLOWER]: Refusing update from remote peer 8cbe7d32e0984b049ca2e3a81d3b5727: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:32.706389 24821 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8cbe7d32e0984b049ca2e3a81d3b5727 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4406ed3b43f1445eb16dc4646496dc79" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:32.711061 24848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4406ed3b43f1445eb16dc4646496dc79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cbe7d32e0984b049ca2e3a81d3b5727. Latest consensus state: current_term: 1 leader_uuid: "8cbe7d32e0984b049ca2e3a81d3b5727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cbe7d32e0984b049ca2e3a81d3b5727" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40551 } } peers { permanent_uuid: "79f1f787f8c541e099b776308ba586b8" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 42295 } } peers { permanent_uuid: "4406ed3b43f1445eb16dc4646496dc79" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 44629 } } }
I20260925 22:45:32.711243 24848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4406ed3b43f1445eb16dc4646496dc79 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:45:32.717387 24822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79f1f787f8c541e099b776308ba586b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cbe7d32e0984b049ca2e3a81d3b5727. Latest consensus state: current_term: 1 leader_uuid: "8cbe7d32e0984b049ca2e3a81d3b5727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cbe7d32e0984b049ca2e3a81d3b5727" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40551 } } peers { permanent_uuid: "79f1f787f8c541e099b776308ba586b8" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 42295 } } peers { permanent_uuid: "4406ed3b43f1445eb16dc4646496dc79" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 44629 } } }
I20260925 22:45:32.717586 24822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79f1f787f8c541e099b776308ba586b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:45:32.729450 24821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cbe7d32e0984b049ca2e3a81d3b5727 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8cbe7d32e0984b049ca2e3a81d3b5727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cbe7d32e0984b049ca2e3a81d3b5727" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40551 } } peers { permanent_uuid: "79f1f787f8c541e099b776308ba586b8" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 42295 } } peers { permanent_uuid: "4406ed3b43f1445eb16dc4646496dc79" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 44629 } } }
I20260925 22:45:32.729606 24821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cbe7d32e0984b049ca2e3a81d3b5727 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:32.730629 24822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79f1f787f8c541e099b776308ba586b8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8cbe7d32e0984b049ca2e3a81d3b5727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cbe7d32e0984b049ca2e3a81d3b5727" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40551 } } peers { permanent_uuid: "79f1f787f8c541e099b776308ba586b8" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 42295 } } peers { permanent_uuid: "4406ed3b43f1445eb16dc4646496dc79" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 44629 } } }
I20260925 22:45:32.730772 24822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79f1f787f8c541e099b776308ba586b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:45:32.732260 24874 mvcc.cc:205] Tried to move back new op lower bound from 7333381458742829056 to 7333381458591059968. Current Snapshot: MvccSnapshot[applied={T|T < 7333381458742829056}]
I20260925 22:45:32.735710 24849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cbe7d32e0984b049ca2e3a81d3b5727 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8cbe7d32e0984b049ca2e3a81d3b5727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cbe7d32e0984b049ca2e3a81d3b5727" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40551 } } peers { permanent_uuid: "79f1f787f8c541e099b776308ba586b8" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 42295 } } peers { permanent_uuid: "4406ed3b43f1445eb16dc4646496dc79" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 44629 } } }
I20260925 22:45:32.735863 24849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cbe7d32e0984b049ca2e3a81d3b5727 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:32.736959 24848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4406ed3b43f1445eb16dc4646496dc79 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8cbe7d32e0984b049ca2e3a81d3b5727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cbe7d32e0984b049ca2e3a81d3b5727" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 40551 } } peers { permanent_uuid: "79f1f787f8c541e099b776308ba586b8" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 42295 } } peers { permanent_uuid: "4406ed3b43f1445eb16dc4646496dc79" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 44629 } } }
I20260925 22:45:32.737094 24848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4406ed3b43f1445eb16dc4646496dc79 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:45:32.738729 24872 mvcc.cc:205] Tried to move back new op lower bound from 7333381458742829056 to 7333381458591059968. Current Snapshot: MvccSnapshot[applied={T|T < 7333381458742829056}]
I20260925 22:45:32.741215 24862 catalog_manager.cc:1347] Generated new cluster ID: 9aa0351424144a1e9d9a581839a46c07
I20260925 22:45:32.741297 24862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:32.796517 24862 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:32.798017 24862 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:32.814575 24862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cbe7d32e0984b049ca2e3a81d3b5727: Generated new TSK 0
I20260925 22:45:32.815559 24862 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:45:32.865556 24818 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:32.862100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58270 (local address 127.2.74.252:44629)
0925 22:45:32.862241 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:32.862247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:32.864353 (+ 2106us) server_negotiation.cc:407] Connection header received
0925 22:45:32.864516 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:32.864520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:32.864579 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:32.864661 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:32.865387 (+ 726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58270: BlockingRecv error: recv got EOF from 127.0.0.1:58270 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 22:45:32.865671 24664 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:32.862520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54380 (local address 127.2.74.254:40551)
0925 22:45:32.862668 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:32.862674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:32.864701 (+ 2027us) server_negotiation.cc:407] Connection header received
0925 22:45:32.864883 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:32.864888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:32.864949 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:32.865006 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:32.865548 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54380: BlockingRecv error: recv got EOF from 127.0.0.1:54380 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 22:45:32.867967 24742 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:32.863723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43076 (local address 127.2.74.253:42295)
0925 22:45:32.863840 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:32.863844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:32.866017 (+ 2173us) server_negotiation.cc:407] Connection header received
0925 22:45:32.866270 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:32.866274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:32.866335 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:32.866442 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:32.867831 (+ 1389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43076: BlockingRecv error: recv got EOF from 127.0.0.1:43076 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 22:45:32.871842 24742 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:32.870055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43086 (local address 127.2.74.253:42295)
0925 22:45:32.870164 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:32.870169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:32.870672 (+ 503us) server_negotiation.cc:407] Connection header received
0925 22:45:32.871003 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:32.871008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:32.871087 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:32.871144 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:32.871713 (+ 569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43086: BlockingRecv error: recv got EOF from 127.0.0.1:43086 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 22:45:32.877321 24742 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:32.875812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43090 (local address 127.2.74.253:42295)
0925 22:45:32.875951 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:32.875956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:32.876240 (+ 284us) server_negotiation.cc:407] Connection header received
0925 22:45:32.876458 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:32.876462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:32.876521 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:32.876576 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:32.877178 (+ 602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43090: BlockingRecv error: recv got EOF from 127.0.0.1:43090 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 22:45:32.883607 24742 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:32.882386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43096 (local address 127.2.74.253:42295)
0925 22:45:32.882527 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:32.882532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:32.882703 (+ 171us) server_negotiation.cc:407] Connection header received
0925 22:45:32.882811 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:32.882814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:32.882878 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:32.883002 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:32.883481 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43096: BlockingRecv error: recv got EOF from 127.0.0.1:43096 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 22:45:32.890810 24742 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:32.889660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43112 (local address 127.2.74.253:42295)
0925 22:45:32.889863 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:32.889868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:32.889982 (+ 114us) server_negotiation.cc:407] Connection header received
0925 22:45:32.890108 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:32.890111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:32.890167 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:32.890267 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:32.890691 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43112: BlockingRecv error: recv got EOF from 127.0.0.1:43112 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
W20260925 22:45:32.902020 24742 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:32.900807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43120 (local address 127.2.74.253:42295)
0925 22:45:32.900953 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:32.900959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:32.901189 (+ 230us) server_negotiation.cc:407] Connection header received
0925 22:45:32.901292 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:32.901294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:32.901345 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:32.901418 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:32.901879 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43120: BlockingRecv error: recv got EOF from 127.0.0.1:43120 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 22:45:32.923760 24742 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:32.922406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43134 (local address 127.2.74.253:42295)
0925 22:45:32.922603 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:32.922611 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:32.922630 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:45:32.922704 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:32.922709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:32.922765 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:32.922871 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:32.923635 (+ 764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43134: BlockingRecv error: recv got EOF from 127.0.0.1:43134 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 22:45:32.958891 24742 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:32.957884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43140 (local address 127.2.74.253:42295)
0925 22:45:32.958026 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:32.958030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:32.958134 (+ 104us) server_negotiation.cc:407] Connection header received
0925 22:45:32.958241 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:32.958244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:32.958290 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:32.958374 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:32.958778 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43140: BlockingRecv error: recv got EOF from 127.0.0.1:43140 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 22:45:33.026521 24742 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:33.025185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43148 (local address 127.2.74.253:42295)
0925 22:45:33.025441 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:33.025452 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:33.025520 (+ 68us) server_negotiation.cc:407] Connection header received
0925 22:45:33.025645 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:33.025648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:33.025746 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:33.025834 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:33.026350 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43148: BlockingRecv error: recv got EOF from 127.0.0.1:43148 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260925 22:45:33.158164 24742 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:33.157005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43162 (local address 127.2.74.253:42295)
0925 22:45:33.157187 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:33.157190 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:33.157236 (+ 46us) server_negotiation.cc:407] Connection header received
0925 22:45:33.157388 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:33.157391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:33.157447 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:33.157525 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:33.158031 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43162: BlockingRecv error: recv got EOF from 127.0.0.1:43162 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260925 22:45:33.417696 24742 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:33.416524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43176 (local address 127.2.74.253:42295)
0925 22:45:33.416671 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:33.416675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:33.416714 (+ 39us) server_negotiation.cc:407] Connection header received
0925 22:45:33.416904 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:33.416906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:33.416954 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:33.417029 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:33.417566 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43176: BlockingRecv error: recv got EOF from 127.0.0.1:43176 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:45:33.641167 24844 catalog_manager.cc:1259] Loaded cluster ID: 9aa0351424144a1e9d9a581839a46c07
I20260925 22:45:33.641264 24844 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 79f1f787f8c541e099b776308ba586b8: loading cluster ID for follower catalog manager: success
I20260925 22:45:33.642923 24844 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 79f1f787f8c541e099b776308ba586b8: acquiring CA information for follower catalog manager: success
I20260925 22:45:33.644160 24844 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 79f1f787f8c541e099b776308ba586b8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:45:33.670322 24860 catalog_manager.cc:1259] Loaded cluster ID: 9aa0351424144a1e9d9a581839a46c07
I20260925 22:45:33.670423 24860 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4406ed3b43f1445eb16dc4646496dc79: loading cluster ID for follower catalog manager: success
I20260925 22:45:33.672227 24860 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4406ed3b43f1445eb16dc4646496dc79: acquiring CA information for follower catalog manager: success
I20260925 22:45:33.673517 24860 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4406ed3b43f1445eb16dc4646496dc79: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:45:33.936010 24945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:33.931923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43186 (local address 127.2.74.253:42295)
0925 22:45:33.932154 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:33.932157 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:33.932174 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:45:33.932354 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:33.932357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:33.932485 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:33.932597 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:45:33.932848 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:33.933363 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:33.934406 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:33.934652 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:33.934765 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:33.935138 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:45:33.935587 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:45:33.935732 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:33.935770 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 22:45:33.935833 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":87,"threads_started":1}
I20260925 22:45:33.943105 24951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:33.939380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58278 (local address 127.2.74.252:44629)
0925 22:45:33.939599 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:33.939603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:33.939617 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:45:33.939751 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:33.939753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:33.939898 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:33.940002 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:45:33.940215 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:33.940729 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:33.941597 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:33.941826 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:33.941918 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:33.942190 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:45:33.942641 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:45:33.942737 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:33.942829 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 22:45:33.942897 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":78,"threads_started":1}
I20260925 22:45:33.950436 24951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:33.946434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58290 (local address 127.2.74.252:44629)
0925 22:45:33.946560 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:33.946566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:33.946907 (+ 341us) server_negotiation.cc:407] Connection header received
0925 22:45:33.947163 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:33.947166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:33.947222 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:33.947316 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:45:33.947666 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:33.948418 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:33.949354 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:33.949599 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:33.949714 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:33.950012 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:45:33.950088 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:45:33.950145 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:33.950270 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 22:45:33.950318 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:45:33.951593 24945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:33.946913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43198 (local address 127.2.74.253:42295)
0925 22:45:33.947075 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:33.947079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:33.947093 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:45:33.947140 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:33.947142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:33.947191 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:33.947271 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:45:33.947546 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:33.948386 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:33.950414 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:33.950727 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:33.950811 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:33.951105 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:45:33.951177 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:45:33.951235 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:33.951383 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 22:45:33.951445 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 22:45:33.952713 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:33.946311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54394 (local address 127.2.74.254:40551)
0925 22:45:33.946560 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:33.946565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:33.946798 (+ 233us) server_negotiation.cc:407] Connection header received
0925 22:45:33.947107 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:33.947110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:33.947289 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:33.947382 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:45:33.947736 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:33.948282 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:33.949857 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:33.950206 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:33.950316 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:33.951842 (+ 1526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:45:33.952223 (+ 381us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:45:33.952304 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:33.952525 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 22:45:33.952593 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":92,"threads_started":1}
I20260925 22:45:33.956353 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 24593
I20260925 22:45:33.964113 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 24668
I20260925 22:45:33.970736 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 24744
2026-09-25T22:45:33Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3209 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:34 dist-test-slave-csnl krb5kdc[24962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:34 dist-test-slave-csnl krb5kdc[24962](info): set up 2 sockets
Sep 25 22:45:34 dist-test-slave-csnl krb5kdc[24962](info): commencing operation
krb5kdc: starting...
W20260925 22:45:35.023958 2347 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/krb5kdc/test-user.keytab.
Sep 25 22:45:35 dist-test-slave-csnl krb5kdc[24962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
Entry for principal kudu/127.2.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab.
I20260925 22:45:35.150329 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:34285
--webserver_interface=127.2.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:34285,127.2.74.253:40653,127.2.74.252:38491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:35.262997 24975 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:45:35.263406 24975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:35.263484 24975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:35.263566 24975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:35.267844 24975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:35.267966 24975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:35.268003 24975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:35.268047 24975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:35.268082 24975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:35.273808 24975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:34285,127.2.74.253:40653,127.2.74.252:38491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.254.keytab
--principal=kudu/127.2.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.254:34285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:35.275111 24975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:35.276273 24975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:35.285375 24980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:35.285385 24981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:35.285517 24975 server_base.cc:1034] running on GCE node
W20260925 22:45:35.285385 24983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:35.286376 24975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:35.287703 24975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:35.288947 24975 hybrid_clock.cc:648] HybridClock initialized: now 1790376335288932 us; error 33 us; skew 500 ppm
Sep 25 22:45:35 dist-test-slave-csnl krb5kdc[24962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376335, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:35.297371 24975 init.cc:377] Logged in from keytab as kudu/127.2.74.254@KRBTEST.COM (short username kudu)
I20260925 22:45:35.298437 24975 webserver.cc:460] Webserver started at http://127.2.74.254:36845/ using document root <none> and password file <none>
I20260925 22:45:35.299134 24975 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:35.299240 24975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:35.299489 24975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:35.301682 24975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-0/data/instance:
uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d"
format_stamp: "Formatted at 2026-09-25 22:45:35 on dist-test-slave-csnl"
server_key: "983d983a40993b51fc3fc8a807246542"
server_key_iv: "16a8572addab5b36c2e2294bfe1f645c"
server_key_version: "encryptionkey@0"
I20260925 22:45:35.302305 24975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-0/wal/instance:
uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d"
format_stamp: "Formatted at 2026-09-25 22:45:35 on dist-test-slave-csnl"
server_key: "983d983a40993b51fc3fc8a807246542"
server_key_iv: "16a8572addab5b36c2e2294bfe1f645c"
server_key_version: "encryptionkey@0"
I20260925 22:45:35.306456 24975 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:45:35.309571 24990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:35.311143 24975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:35.311318 24975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-0/wal
uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d"
format_stamp: "Formatted at 2026-09-25 22:45:35 on dist-test-slave-csnl"
server_key: "983d983a40993b51fc3fc8a807246542"
server_key_iv: "16a8572addab5b36c2e2294bfe1f645c"
server_key_version: "encryptionkey@0"
I20260925 22:45:35.311467 24975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:35.335814 24975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:35.339337 24975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:35.339555 24975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:35.347206 24975 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.254:34285
I20260925 22:45:35.347537 25042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.254:34285 every 8 connection(s)
I20260925 22:45:35.348369 24975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:35.351446 25043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:35.355547 25043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34285 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:45:35.357982 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 24975
I20260925 22:45:35.358134 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-0/wal/instance
I20260925 22:45:35.358449 2347 external_mini_cluster.cc:1428] Setting key b217b2106ab3117bd615e2822d0e4f68
WARNING: no policy specified for kudu/127.2.74.253@KRBTEST.COM; defaulting to no policy
Sep 25 22:45:35 dist-test-slave-csnl krb5kdc[24962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376335, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.253@KRBTEST.COM" created.
I20260925 22:45:35.373402 25045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.357947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35664 (local address 127.2.74.254:34285)
0925 22:45:35.358659 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:35.358672 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:35.358831 (+ 159us) server_negotiation.cc:407] Connection header received
0925 22:45:35.360004 (+ 1173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:35.360015 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:35.360273 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:35.360441 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:35.361632 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.363159 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.364298 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.364983 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.365184 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:35.367957 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:35.367965 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:35.367976 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:35.368001 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:35.369640 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:35.370315 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:35.370324 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:35.370329 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:35.370577 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:35.371061 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:35.371064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:35.371065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:35.371504 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:35.371619 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:35.372186 (+ 567us) server_negotiation.cc:299] Negotiation successful
0925 22:45:35.372519 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":588,"spinlock_wait_cycles":5504,"thread_start_us":232,"threads_started":1}
I20260925 22:45:35.373405 25046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.357946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34285 (local address 127.0.0.1:35664)
0925 22:45:35.358590 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:35.358652 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:35.359625 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:35.360506 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:35.360515 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:35.361078 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:35.361449 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:35.361466 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.363342 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.363344 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:35.364133 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:35.364144 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.365178 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.365181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:35.365305 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:35.365980 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:35.365998 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:35.367763 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:35.369813 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:35.369822 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:35.369831 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:35.370181 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:35.370709 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:35.370716 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:35.370721 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:35.370889 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:35.371632 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:35.371643 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:35.371846 (+ 203us) client_negotiation.cc:769] Sending connection context
0925 22:45:35.372051 (+ 205us) client_negotiation.cc:240] Negotiation successful
0925 22:45:35.372370 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":1,"spinlock_wait_cycles":60288,"thread_start_us":225,"threads_started":1}
I20260925 22:45:35.376188 25043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.253" port: 40653 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:45:35.377110 25046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.376737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:40653 (local address 127.0.0.1:47348)
0925 22:45:35.376898 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:35.377007 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.253:40653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 22:45:35.377369 24994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.74.253:40653: connect: Connection refused (error 111)
I20260925 22:45:35.378355 25046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.378059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:40653 (local address 127.0.0.1:47358)
0925 22:45:35.378210 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:35.378275 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.253:40653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 22:45:35.378588 25043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.74.253:40653: Network error: Client connection negotiation failed: client connection to 127.2.74.253:40653: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.253.keytab.
Entry for principal kudu/127.2.74.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.253.keytab.
I20260925 22:45:35.389804 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.253:40653
--webserver_interface=127.2.74.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.253.keytab
--principal=kudu/127.2.74.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:34285,127.2.74.253:40653,127.2.74.252:38491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:45:35.427906 25043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.74.253" port: 40653 } attempt: 1
I20260925 22:45:35.429040 25046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.428615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:40653 (local address 127.0.0.1:47368)
0925 22:45:35.428858 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:35.428942 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.253:40653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 22:45:35.429893 25046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.429618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:40653 (local address 127.0.0.1:47384)
0925 22:45:35.429752 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:35.429813 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.253:40653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 22:45:35.430044 25043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.74.253:40653: Network error: Client connection negotiation failed: client connection to 127.2.74.253:40653: connect: Connection refused (error 111)
I20260925 22:45:35.498242 25043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.74.253" port: 40653 } attempt: 2
I20260925 22:45:35.499244 25046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.498882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:40653 (local address 127.0.0.1:47392)
0925 22:45:35.499078 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:35.499148 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.253:40653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 22:45:35.500057 25046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.499795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:40653 (local address 127.0.0.1:47394)
0925 22:45:35.499927 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:35.499989 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.253:40653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 22:45:35.500262 25043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.74.253:40653: Network error: Client connection negotiation failed: client connection to 127.2.74.253:40653: connect: Connection refused (error 111)
W20260925 22:45:35.509467 25050 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:45:35.509879 25050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:35.509943 25050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:35.510022 25050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:35.514394 25050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:35.514516 25050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:35.514547 25050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:35.514576 25050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:35.514607 25050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:35.520735 25050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:34285,127.2.74.253:40653,127.2.74.252:38491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.253.keytab
--principal=kudu/127.2.74.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.253:40653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.74.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:35.522020 25050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:35.523202 25050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:35.532339 25056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:35.532342 25058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:35.532342 25055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:35.532795 25050 server_base.cc:1034] running on GCE node
I20260925 22:45:35.533473 25050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:35.534706 25050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:35.535904 25050 hybrid_clock.cc:648] HybridClock initialized: now 1790376335535875 us; error 46 us; skew 500 ppm
Sep 25 22:45:35 dist-test-slave-csnl krb5kdc[24962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376335, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:35.538877 25050 init.cc:377] Logged in from keytab as kudu/127.2.74.253@KRBTEST.COM (short username kudu)
I20260925 22:45:35.539882 25050 webserver.cc:460] Webserver started at http://127.2.74.253:41803/ using document root <none> and password file <none>
I20260925 22:45:35.540438 25050 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:35.540503 25050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:35.540688 25050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:35.542826 25050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-1/data/instance:
uuid: "90787e1963564cbb8944278903bd2b64"
format_stamp: "Formatted at 2026-09-25 22:45:35 on dist-test-slave-csnl"
server_key: "94f7eecda4d6be092ad962aac2d4addf"
server_key_iv: "df832e725d1b38cb5e1d7aa7bcfabeb5"
server_key_version: "encryptionkey@0"
I20260925 22:45:35.543488 25050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-1/wal/instance:
uuid: "90787e1963564cbb8944278903bd2b64"
format_stamp: "Formatted at 2026-09-25 22:45:35 on dist-test-slave-csnl"
server_key: "94f7eecda4d6be092ad962aac2d4addf"
server_key_iv: "df832e725d1b38cb5e1d7aa7bcfabeb5"
server_key_version: "encryptionkey@0"
I20260925 22:45:35.547610 25050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:45:35.550503 25065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:35.551954 25050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:35.552126 25050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-1/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-1/wal
uuid: "90787e1963564cbb8944278903bd2b64"
format_stamp: "Formatted at 2026-09-25 22:45:35 on dist-test-slave-csnl"
server_key: "94f7eecda4d6be092ad962aac2d4addf"
server_key_iv: "df832e725d1b38cb5e1d7aa7bcfabeb5"
server_key_version: "encryptionkey@0"
I20260925 22:45:35.552271 25050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:35.580384 25050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:35.583858 25050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:35.584097 25050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:35.591464 25043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.74.253" port: 40653 } attempt: 3
I20260925 22:45:35.591884 25050 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.253:40653
I20260925 22:45:35.592234 25117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.253:40653 every 8 connection(s)
I20260925 22:45:35.593044 25050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-1/data/info.pb
I20260925 22:45:35.596088 25118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:35.597393 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 25050
I20260925 22:45:35.597548 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-1/wal/instance
I20260925 22:45:35.597841 2347 external_mini_cluster.cc:1428] Setting key beddc4e78efc942300f34880e8fe87f5
Sep 25 22:45:35 dist-test-slave-csnl krb5kdc[24962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376335, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for kudu/127.2.74.253@KRBTEST.COM
I20260925 22:45:35.601145 25118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34285 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:45:35.606654 25046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.592072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:40653 (local address 127.0.0.1:47402)
0925 22:45:35.592270 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:35.592289 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:35.592408 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:35.595417 (+ 3009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:35.595421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:35.595650 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:35.595801 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:35.595810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.597328 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.597333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:35.598159 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:35.598174 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.599389 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.599394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:35.599513 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:35.600161 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:35.600178 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:35.601709 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:35.604366 (+ 2657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:35.604370 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:35.604372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:35.604731 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:35.605121 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:35.605124 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:35.605126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:35.605228 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:35.606036 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:35.606040 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:35.606163 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 22:45:35.606295 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 22:45:35.606458 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
WARNING: no policy specified for kudu/127.2.74.252@KRBTEST.COM; defaulting to no policy
I20260925 22:45:35.608130 25120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.593323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47402 (local address 127.2.74.253:40653)
0925 22:45:35.593880 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:35.593897 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:35.593937 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:45:35.594610 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:35.594627 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:35.595089 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:35.595475 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:35.595981 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.597137 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.598326 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.599224 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.599435 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:35.601961 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:35.601967 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:35.601984 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:35.602015 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:35.604219 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:35.604872 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:35.604877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:35.604882 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:35.605007 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:35.605328 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:35.605332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:35.605334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:35.605879 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:35.606017 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:35.606469 (+ 452us) server_negotiation.cc:299] Negotiation successful
0925 22:45:35.606810 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":402,"thread_start_us":176,"threads_started":1}
I20260925 22:45:35.610576 25043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.252" port: 38491 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.74.252@KRBTEST.COM" created.
Sep 25 22:45:35 dist-test-slave-csnl krb5kdc[24962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376335, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.253@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:45:35.611438 25046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.611175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:38491 (local address 127.0.0.1:51622)
0925 22:45:35.611293 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:35.611353 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 22:45:35.612390 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.611926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:38491 (local address 127.0.0.1:51626)
0925 22:45:35.612244 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:35.612302 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":114,"threads_started":1}
W20260925 22:45:35.612816 25043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.74.252:38491: Network error: Client connection negotiation failed: client connection to 127.2.74.252:38491: connect: Connection refused (error 111)
I20260925 22:45:35.616179 25122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.603617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34285 (local address 127.0.0.1:35668)
0925 22:45:35.604137 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:35.604204 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:35.604434 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:35.604832 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:35.604843 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:35.605352 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:35.605589 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:35.605601 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.606608 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.606611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:35.607564 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:35.607576 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.608905 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.608908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:35.609035 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:35.609692 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:35.609705 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:35.611743 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:35.613939 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:35.613952 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:35.613960 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:35.614241 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:35.614607 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:35.614610 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:35.614612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:35.614735 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:35.615312 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:35.615322 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:35.615495 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 22:45:35.615753 (+ 258us) client_negotiation.cc:240] Negotiation successful
0925 22:45:35.615977 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":3,"thread_start_us":168,"threads_started":1}
I20260925 22:45:35.616165 25045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.603723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35668 (local address 127.2.74.254:34285)
0925 22:45:35.603875 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:35.603882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:35.604311 (+ 429us) server_negotiation.cc:407] Connection header received
0925 22:45:35.604569 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:35.604572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:35.604635 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:35.604724 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:35.605739 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.606470 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.607704 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.608669 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.608782 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:35.611913 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:35.611920 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:35.611926 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:35.611961 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:35.613741 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:35.614357 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:35.614364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:35.614369 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:35.614469 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:35.614855 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:35.614861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:35.614866 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:35.615177 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:35.615303 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:35.615812 (+ 509us) server_negotiation.cc:299] Negotiation successful
0925 22:45:35.615963 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:45:35.617796 25118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.253" port: 40653 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:45:35 dist-test-slave-csnl krb5kdc[24962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376335, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.253@KRBTEST.COM for kudu/127.2.74.253@KRBTEST.COM
I20260925 22:45:35.627429 25122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.618406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:40653 (local address 127.0.0.1:47416)
0925 22:45:35.618530 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:35.618545 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:35.618630 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:35.618924 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:35.618927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:35.619175 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:35.619292 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:35.619298 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.620418 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.620421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:35.621014 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:35.621022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.621557 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.621561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:35.621681 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:35.622254 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:35.622271 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:35.623889 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:35.625753 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:35.625757 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:35.625758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:35.625977 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:35.626279 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:35.626281 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:35.626283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:35.626368 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:35.626749 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:35.626751 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:35.626843 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 22:45:35.627123 (+ 280us) client_negotiation.cc:240] Negotiation successful
0925 22:45:35.627270 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 22:45:35.627456 25120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.618469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47416 (local address 127.2.74.253:40653)
0925 22:45:35.618638 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:35.618645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:35.618660 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:45:35.618724 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:35.618727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:35.618781 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:35.618879 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:35.619426 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.620218 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.621145 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.621391 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.621488 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:35.624084 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:35.624087 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:35.624089 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:35.624111 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:35.625582 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:35.626095 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:35.626098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:35.626099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:35.626194 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:35.626449 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:35.626452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:35.626453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:35.626632 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:35.626717 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:35.627160 (+ 443us) server_negotiation.cc:299] Negotiation successful
0925 22:45:35.627329 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":53}
I20260925 22:45:35.628383 25118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.252" port: 38491 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:45:35.629237 25122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.628900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:38491 (local address 127.0.0.1:51638)
0925 22:45:35.629039 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:35.629175 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 22:45:35.629938 25068 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.74.252:38491: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.74.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.252.keytab.
Entry for principal kudu/127.2.74.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.252.keytab.
I20260925 22:45:35.631093 2347 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
/tmp/dist-test-taskx4wBtU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.252:38491
--webserver_interface=127.2.74.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.252.keytab
--principal=kudu/127.2.74.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.74.212:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.74.254:34285,127.2.74.253:40653,127.2.74.252:38491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:45:35.631358 25122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.630883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:38491 (local address 127.0.0.1:51644)
0925 22:45:35.631040 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:35.631096 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260925 22:45:35.631564 25118 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.74.252:38491: Network error: Client connection negotiation failed: client connection to 127.2.74.252:38491: connect: Connection refused (error 111)
I20260925 22:45:35.644057 25043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.74.252" port: 38491 } attempt: 1
I20260925 22:45:35.645073 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.644691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:38491 (local address 127.0.0.1:51646)
0925 22:45:35.644887 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:35.644979 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 22:45:35.646144 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.645827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:38491 (local address 127.0.0.1:51650)
0925 22:45:35.646005 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:35.646068 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260925 22:45:35.646340 25043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.74.252:38491: Network error: Client connection negotiation failed: client connection to 127.2.74.252:38491: connect: Connection refused (error 111)
I20260925 22:45:35.680944 25118 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.74.252" port: 38491 } attempt: 1
I20260925 22:45:35.681929 25122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.681590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:38491 (local address 127.0.0.1:51662)
0925 22:45:35.681758 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:35.681835 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 22:45:35.682808 25122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.682561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:38491 (local address 127.0.0.1:51674)
0925 22:45:35.682677 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:35.682739 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 22:45:35.683017 25118 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.74.252:38491: Network error: Client connection negotiation failed: client connection to 127.2.74.252:38491: connect: Connection refused (error 111)
I20260925 22:45:35.721572 25043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.74.252" port: 38491 } attempt: 2
I20260925 22:45:35.722635 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.722235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:38491 (local address 127.0.0.1:51678)
0925 22:45:35.722449 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:35.722532 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 22:45:35.723677 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.723371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:38491 (local address 127.0.0.1:51694)
0925 22:45:35.723516 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:35.723581 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 22:45:35.723884 25043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.74.252:38491: Network error: Client connection negotiation failed: client connection to 127.2.74.252:38491: connect: Connection refused (error 111)
I20260925 22:45:35.751204 25118 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.74.252" port: 38491 } attempt: 2
I20260925 22:45:35.752266 25122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.751940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:38491 (local address 127.0.0.1:51710)
0925 22:45:35.752110 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:35.752180 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 22:45:35.753095 25122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.752857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:38491 (local address 127.0.0.1:51716)
0925 22:45:35.752973 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:35.753031 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 22:45:35.753223 25118 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.74.252:38491: Network error: Client connection negotiation failed: client connection to 127.2.74.252:38491: connect: Connection refused (error 111)
W20260925 22:45:35.754351 25126 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:45:35.754751 25126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:35.754817 25126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:35.754899 25126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:35.759454 25126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:35.759573 25126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:35.759608 25126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:35.759647 25126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:35.759714 25126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:35.765843 25126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.74.212:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.74.254:34285,127.2.74.253:40653,127.2.74.252:38491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/krb5kdc/kudu_127.2.74.252.keytab
--principal=kudu/127.2.74.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.74.252:38491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.74.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:35.767248 25126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:35.768409 25126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:35.778066 25132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:35.778069 25134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:35.778074 25131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:35.778568 25126 server_base.cc:1034] running on GCE node
I20260925 22:45:35.779438 25126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:35.780786 25126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:35.782006 25126 hybrid_clock.cc:648] HybridClock initialized: now 1790376335781987 us; error 39 us; skew 500 ppm
Sep 25 22:45:35 dist-test-slave-csnl krb5kdc[24962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376335, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:35.785463 25126 init.cc:377] Logged in from keytab as kudu/127.2.74.252@KRBTEST.COM (short username kudu)
I20260925 22:45:35.786624 25126 webserver.cc:460] Webserver started at http://127.2.74.252:33503/ using document root <none> and password file <none>
I20260925 22:45:35.787351 25126 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:35.787472 25126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:35.787729 25126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:35.790114 25126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-2/data/instance:
uuid: "fe09e14feb0845b5ab2bd4173efa7f54"
format_stamp: "Formatted at 2026-09-25 22:45:35 on dist-test-slave-csnl"
server_key: "2bc3f2124d2dc93bc23ec7434ab268dd"
server_key_iv: "2737f7b252a7a5691e3e13fde27bb3e7"
server_key_version: "encryptionkey@0"
I20260925 22:45:35.790782 25126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-2/wal/instance:
uuid: "fe09e14feb0845b5ab2bd4173efa7f54"
format_stamp: "Formatted at 2026-09-25 22:45:35 on dist-test-slave-csnl"
server_key: "2bc3f2124d2dc93bc23ec7434ab268dd"
server_key_iv: "2737f7b252a7a5691e3e13fde27bb3e7"
server_key_version: "encryptionkey@0"
I20260925 22:45:35.795245 25126 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:45:35.798555 25141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:35.800554 25126 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:45:35.800820 25126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-2/data,/tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-2/wal
uuid: "fe09e14feb0845b5ab2bd4173efa7f54"
format_stamp: "Formatted at 2026-09-25 22:45:35 on dist-test-slave-csnl"
server_key: "2bc3f2124d2dc93bc23ec7434ab268dd"
server_key_iv: "2737f7b252a7a5691e3e13fde27bb3e7"
server_key_version: "encryptionkey@0"
I20260925 22:45:35.800987 25126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:35.822297 25126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:35.823055 25043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.74.252" port: 38491 } attempt: 3
I20260925 22:45:35.823938 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.823589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:38491 (local address 127.0.0.1:51728)
0925 22:45:35.823761 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:35.823832 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 22:45:35.824747 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.824509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:38491 (local address 127.0.0.1:51730)
0925 22:45:35.824631 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:35.824686 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.74.252:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 22:45:35.824959 25043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.74.252:38491: Network error: Client connection negotiation failed: client connection to 127.2.74.252:38491: connect: Connection refused (error 111)
I20260925 22:45:35.826509 25126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:35.826740 25126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:35.834345 25126 rpc_server.cc:307] RPC server started. Bound to: 127.2.74.252:38491
I20260925 22:45:35.834666 25193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.74.252:38491 every 8 connection(s)
I20260925 22:45:35.835507 25126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-2/data/info.pb
I20260925 22:45:35.838490 25194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:35.839612 2347 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu as pid 25126
I20260925 22:45:35.839768 2347 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4wBtU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376047580784-2347-0/minicluster-data/master-2/wal/instance
I20260925 22:45:35.840094 2347 external_mini_cluster.cc:1428] Setting key 01e9d8386707e311e814ed69609842f7
I20260925 22:45:35.843360 25194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34285 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:45:35.844390 25118 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.74.252" port: 38491 } attempt: 3
Sep 25 22:45:35 dist-test-slave-csnl krb5kdc[24962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376335, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.253@KRBTEST.COM for kudu/127.2.74.252@KRBTEST.COM
Sep 25 22:45:35 dist-test-slave-csnl krb5kdc[24962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.252@KRBTEST.COM
Sep 25 22:45:35 dist-test-slave-csnl krb5kdc[24962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376335, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.252@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:45:35.856192 25122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.844922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:38491 (local address 127.0.0.1:51734)
0925 22:45:35.845127 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:35.845143 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:35.845224 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:35.845660 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:35.845663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:35.845887 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:35.845999 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:35.846007 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.847063 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.847067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:35.847621 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:35.847635 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.848927 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.848931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:35.849024 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:35.849544 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:35.849560 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:35.851003 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:35.853686 (+ 2683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:35.853692 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:35.853696 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:35.853984 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:35.854562 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:35.854565 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:35.854567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:35.854665 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:35.855587 (+ 922us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:35.855590 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:35.855682 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 22:45:35.855821 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 22:45:35.855990 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260925 22:45:35.857398 25045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.845820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35680 (local address 127.2.74.254:34285)
0925 22:45:35.846003 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:35.846008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:35.846374 (+ 366us) server_negotiation.cc:407] Connection header received
0925 22:45:35.846624 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:35.846628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:35.846679 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:35.846757 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:35.847662 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.848176 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.849180 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.849552 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.849671 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:35.853263 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:35.853267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:35.853269 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:35.853313 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:35.855013 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:35.855763 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:35.855765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:35.855767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:35.855891 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:35.856232 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:35.856236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:35.856238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:35.856535 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:35.856611 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:35.857070 (+ 459us) server_negotiation.cc:299] Negotiation successful
0925 22:45:35.857193 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260925 22:45:35.857519 25197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.841584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51732 (local address 127.2.74.252:38491)
0925 22:45:35.842059 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:35.842082 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:35.842130 (+ 48us) server_negotiation.cc:407] Connection header received
0925 22:45:35.842837 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:35.842863 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:35.843341 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:35.843667 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:35.844750 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.846363 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.847663 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.848761 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.848940 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:35.851891 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:35.851895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:35.851898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:35.851928 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:35.854703 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:35.855295 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:35.855298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:35.855301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:35.855418 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:35.855759 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:35.855762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:35.855765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:35.855972 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:35.856796 (+ 824us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:35.856877 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 22:45:35.856989 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":128,"server-negotiator.queue_time_us":318,"spinlock_wait_cycles":17920,"thread_start_us":153,"threads_started":1}
I20260925 22:45:35.858171 25199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.845555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.254:34285 (local address 127.0.0.1:35680)
0925 22:45:35.846235 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:35.846288 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:35.846500 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:35.846838 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:35.846845 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:35.847366 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:35.847517 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:35.847524 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.848292 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.848297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:35.848994 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:35.849006 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.850576 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.850579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:35.850730 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:35.851365 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:35.851378 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:35.853095 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:35.855100 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:35.855111 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:35.855118 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:35.855418 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:35.856002 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:35.856005 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:35.856007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:35.856090 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:35.856706 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:35.856712 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:35.856829 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 22:45:35.857705 (+ 876us) client_negotiation.cc:240] Negotiation successful
0925 22:45:35.857939 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"spinlock_wait_cycles":2304,"thread_start_us":147,"threads_started":1}
I20260925 22:45:35.857518 25198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.844980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51734 (local address 127.2.74.252:38491)
0925 22:45:35.845353 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:35.845364 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:35.845380 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:45:35.845422 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:35.845428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:35.845476 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:35.845615 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:35.846133 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.846902 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.847746 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.848761 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.849284 (+ 523us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:35.851176 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:35.851186 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:35.851203 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:35.851242 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:35.853556 (+ 2314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:35.854303 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:35.854310 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:35.854317 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:35.854450 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:35.854769 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:35.854773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:35.854775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:35.855479 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:35.855623 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:35.856012 (+ 389us) server_negotiation.cc:299] Negotiation successful
0925 22:45:35.856407 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":147,"server-negotiator.queue_time_us":289,"thread_start_us":119,"threads_started":1}
I20260925 22:45:35.859782 25194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.253" port: 40653 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:45:35.862704 25118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90787e1963564cbb8944278903bd2b64: Bootstrap starting.
I20260925 22:45:35.865607 25118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90787e1963564cbb8944278903bd2b64: Neither blocks nor log segments found. Creating new log.
Sep 25 22:45:35 dist-test-slave-csnl krb5kdc[24962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376335, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.252@KRBTEST.COM for kudu/127.2.74.253@KRBTEST.COM
Sep 25 22:45:35 dist-test-slave-csnl krb5kdc[24962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.254@KRBTEST.COM
I20260925 22:45:35.866550 25118 log.cc:824] T 00000000000000000000000000000000 P 90787e1963564cbb8944278903bd2b64: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:35.869690 25118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90787e1963564cbb8944278903bd2b64: No bootstrap required, opened a new log
I20260925 22:45:35.870333 25120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.860502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47422 (local address 127.2.74.253:40653)
0925 22:45:35.860612 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:35.860616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:35.860688 (+ 72us) server_negotiation.cc:407] Connection header received
0925 22:45:35.860865 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:35.860870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:35.860928 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:35.860984 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:35.861620 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.862205 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.863245 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.863586 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.863708 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:35.866308 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:35.866314 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:35.866318 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:35.866345 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:35.868483 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:35.868985 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:35.868988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:35.868990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:35.869099 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:35.869385 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:35.869388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:35.869390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:35.869650 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:35.869845 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:35.870062 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 22:45:35.870176 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 22:45:35.870333 25199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.860265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.253:40653 (local address 127.0.0.1:47422)
0925 22:45:35.860629 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:35.860645 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:35.860735 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:35.861160 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:35.861163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:35.861385 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:35.861498 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:35.861504 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.862322 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.862325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:35.863117 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:35.863130 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.863713 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.863717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:35.863819 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:35.864494 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:35.864510 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:35.866136 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:35.868613 (+ 2477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:35.868616 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:35.868618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:35.868872 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:35.869189 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:35.869192 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:35.869194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:35.869296 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:35.869751 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:35.869754 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:35.869877 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 22:45:35.869999 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 22:45:35.870164 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263}
I20260925 22:45:35.870935 25045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.860987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35686 (local address 127.2.74.254:34285)
0925 22:45:35.861110 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:35.861114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:35.861127 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:45:35.861159 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:35.861162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:35.861205 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:35.861284 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:35.862075 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.862647 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.863756 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.864027 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.864157 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:35.866883 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:35.866887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:35.866890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:35.866922 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:35.868602 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:35.869115 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:35.869121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:35.869125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:35.869217 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:35.869533 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:35.869537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:35.869538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:35.869768 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:35.870014 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:35.870652 (+ 638us) server_negotiation.cc:299] Negotiation successful
0925 22:45:35.870777 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:45:35.871578 25194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.74.252" port: 38491 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:45:35.874634 25118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90787e1963564cbb8944278903bd2b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34285 } } peers { permanent_uuid: "90787e1963564cbb8944278903bd2b64" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 40653 } } peers { permanent_uuid: "fe09e14feb0845b5ab2bd4173efa7f54" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 38491 } }
I20260925 22:45:35.874927 25118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90787e1963564cbb8944278903bd2b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90787e1963564cbb8944278903bd2b64, State: Initialized, Role: FOLLOWER
I20260925 22:45:35.875708 25118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90787e1963564cbb8944278903bd2b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34285 } } peers { permanent_uuid: "90787e1963564cbb8944278903bd2b64" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 40653 } } peers { permanent_uuid: "fe09e14feb0845b5ab2bd4173efa7f54" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 38491 } }
I20260925 22:45:35.876276 25201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90787e1963564cbb8944278903bd2b64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34285 } } peers { permanent_uuid: "90787e1963564cbb8944278903bd2b64" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 40653 } } peers { permanent_uuid: "fe09e14feb0845b5ab2bd4173efa7f54" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 38491 } } }
I20260925 22:45:35.876410 25201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90787e1963564cbb8944278903bd2b64 [sys.catalog]: This master's current role is: FOLLOWER
Sep 25 22:45:35 dist-test-slave-csnl krb5kdc[24962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376335, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.252@KRBTEST.COM for kudu/127.2.74.252@KRBTEST.COM
I20260925 22:45:35.877468 25118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90787e1963564cbb8944278903bd2b64 [sys.catalog]: configured and running, proceeding with master startup.
Sep 25 22:45:35 dist-test-slave-csnl krb5kdc[24962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.74.253@KRBTEST.COM
I20260925 22:45:35.881230 25199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.872131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:38491 (local address 127.0.0.1:51736)
0925 22:45:35.872243 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:35.872257 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:35.872346 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:35.872589 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:35.872591 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:35.872768 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:35.872835 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:35.872840 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.873741 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.873744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:35.874254 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:35.874261 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.874740 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.874743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:35.874810 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:35.875351 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:35.875362 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:35.876825 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:35.878984 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:35.878988 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:35.878990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:35.879245 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:35.879561 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:35.879564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:35.879566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:35.879663 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:35.880707 (+ 1044us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:35.880710 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:35.880803 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 22:45:35.880919 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 22:45:35.881065 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 22:45:35.881876 25198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.872181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51736 (local address 127.2.74.252:38491)
0925 22:45:35.872274 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:35.872281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:35.872295 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:45:35.872456 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:35.872458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:35.872495 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:35.872580 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:35.872940 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.873629 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.874375 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.874631 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.874741 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:35.876979 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:35.876983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:35.876985 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:35.877011 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:35.878792 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:35.879355 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:35.879361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:35.879363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:35.879449 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:35.879747 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:35.879750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:35.879751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:35.880329 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:35.880665 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:35.881607 (+ 942us) server_negotiation.cc:299] Negotiation successful
0925 22:45:35.881734 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 22:45:35.884107 25120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.872967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47432 (local address 127.2.74.253:40653)
0925 22:45:35.873076 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:35.873080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:35.873093 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:45:35.873183 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:35.873186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:35.873229 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:35.873300 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:35.874082 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.874766 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.875947 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.876324 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.876856 (+ 532us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:35.879602 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:35.879608 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:35.879614 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:35.879647 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:35.881241 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:35.881743 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:35.881747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:35.881748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:35.881838 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:35.882420 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:35.882424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:35.882426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:35.882637 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:35.882809 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:35.883776 (+ 967us) server_negotiation.cc:299] Negotiation successful
0925 22:45:35.883935 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 22:45:35.885300 25194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe09e14feb0845b5ab2bd4173efa7f54: Bootstrap starting.
I20260925 22:45:35.882258 25050 master_runner.cc:186] Error getting master registration for 127.2.74.254:34285: OK, instance_id { permanent_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d" instance_seqno: 1790376335344947 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:45:35.885895 25050 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:45:35.887900 25194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe09e14feb0845b5ab2bd4173efa7f54: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:35.888790 25194 log.cc:824] T 00000000000000000000000000000000 P fe09e14feb0845b5ab2bd4173efa7f54: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:35.889549 2347 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 22:45:35.889693 25212 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 90787e1963564cbb8944278903bd2b64: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:45:35.889775 25212 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:45:35.892192 25194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe09e14feb0845b5ab2bd4173efa7f54: No bootstrap required, opened a new log
I20260925 22:45:35.890993 25017 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35686
I20260925 22:45:35.895488 25194 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe09e14feb0845b5ab2bd4173efa7f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34285 } } peers { permanent_uuid: "90787e1963564cbb8944278903bd2b64" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 40653 } } peers { permanent_uuid: "fe09e14feb0845b5ab2bd4173efa7f54" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 38491 } }
I20260925 22:45:35.895757 25194 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe09e14feb0845b5ab2bd4173efa7f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe09e14feb0845b5ab2bd4173efa7f54, State: Initialized, Role: FOLLOWER
I20260925 22:45:35.896097 25194 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe09e14feb0845b5ab2bd4173efa7f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34285 } } peers { permanent_uuid: "90787e1963564cbb8944278903bd2b64" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 40653 } } peers { permanent_uuid: "fe09e14feb0845b5ab2bd4173efa7f54" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 38491 } }
I20260925 22:45:35.896782 25215 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe09e14feb0845b5ab2bd4173efa7f54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34285 } } peers { permanent_uuid: "90787e1963564cbb8944278903bd2b64" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 40653 } } peers { permanent_uuid: "fe09e14feb0845b5ab2bd4173efa7f54" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 38491 } } }
I20260925 22:45:35.897050 25215 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe09e14feb0845b5ab2bd4173efa7f54 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:45:35.897536 25194 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe09e14feb0845b5ab2bd4173efa7f54 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 22:45:35.903299 25226 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fe09e14feb0845b5ab2bd4173efa7f54: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:45:35.903388 25226 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:45:35.901523 25126 master_runner.cc:186] Error getting master registration for 127.2.74.254:34285: OK, instance_id { permanent_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d" instance_seqno: 1790376335344947 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:45:35.905189 25126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:45:35.989147 25043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.74.252" port: 38491 } attempt: 4
Sep 25 22:45:35 dist-test-slave-csnl krb5kdc[24962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376335, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.74.254@KRBTEST.COM for kudu/127.2.74.252@KRBTEST.COM
I20260925 22:45:36.001802 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.989805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.74.252:38491 (local address 127.0.0.1:51748)
0925 22:45:35.989997 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:35.990016 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:35.990144 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:35.990715 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:35.990719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:35.991019 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:35.991187 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:35.991196 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.992370 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.992374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:35.993344 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:35.993360 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.994044 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.994048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:35.994149 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:35.994828 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:35.994845 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:35.996686 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:35.999486 (+ 2800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:35.999491 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:35.999493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:35.999888 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:36.000201 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:36.000203 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:36.000205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:36.000295 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:36.001157 (+ 862us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:36.001160 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:36.001297 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 22:45:36.001413 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 22:45:36.001591 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 22:45:36.001806 25198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:35.990320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51748 (local address 127.2.74.252:38491)
0925 22:45:35.990456 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:35.990461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:35.990476 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:45:35.990522 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:35.990525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:35.990578 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:35.990671 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:35.991358 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.992220 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.993540 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:35.993907 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:35.994028 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:35.996892 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:35.996897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:35.996900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:35.996935 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:35.999332 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:35.999997 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:36.000001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:36.000004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:36.000106 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:36.000389 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:36.000392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:36.000394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:36.000883 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:36.001205 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:36.001503 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 22:45:36.001651 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 22:45:36.006002 25043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe55c1f2280d41aabd0d3a40cfddbb1d: Bootstrap starting.
I20260925 22:45:36.009377 25043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe55c1f2280d41aabd0d3a40cfddbb1d: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:36.010438 25043 log.cc:824] T 00000000000000000000000000000000 P fe55c1f2280d41aabd0d3a40cfddbb1d: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:36.013274 25043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe55c1f2280d41aabd0d3a40cfddbb1d: No bootstrap required, opened a new log
I20260925 22:45:36.018291 25043 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe55c1f2280d41aabd0d3a40cfddbb1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34285 } } peers { permanent_uuid: "90787e1963564cbb8944278903bd2b64" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 40653 } } peers { permanent_uuid: "fe09e14feb0845b5ab2bd4173efa7f54" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 38491 } }
I20260925 22:45:36.018762 25043 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe55c1f2280d41aabd0d3a40cfddbb1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe55c1f2280d41aabd0d3a40cfddbb1d, State: Initialized, Role: FOLLOWER
I20260925 22:45:36.019277 25043 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe55c1f2280d41aabd0d3a40cfddbb1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34285 } } peers { permanent_uuid: "90787e1963564cbb8944278903bd2b64" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 40653 } } peers { permanent_uuid: "fe09e14feb0845b5ab2bd4173efa7f54" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 38491 } }
I20260925 22:45:36.020498 25229 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe55c1f2280d41aabd0d3a40cfddbb1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34285 } } peers { permanent_uuid: "90787e1963564cbb8944278903bd2b64" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 40653 } } peers { permanent_uuid: "fe09e14feb0845b5ab2bd4173efa7f54" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 38491 } } }
I20260925 22:45:36.021023 25229 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe55c1f2280d41aabd0d3a40cfddbb1d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:45:36.021697 25043 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe55c1f2280d41aabd0d3a40cfddbb1d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:36.021772 25017 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe55c1f2280d41aabd0d3a40cfddbb1d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 22:45:36.022047 25017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe55c1f2280d41aabd0d3a40cfddbb1d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:36.023607 25017 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe55c1f2280d41aabd0d3a40cfddbb1d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34285 } } peers { permanent_uuid: "90787e1963564cbb8944278903bd2b64" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 40653 } } peers { permanent_uuid: "fe09e14feb0845b5ab2bd4173efa7f54" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 38491 } }
I20260925 22:45:36.025758 25168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fe09e14feb0845b5ab2bd4173efa7f54"
I20260925 22:45:36.026052 25168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe09e14feb0845b5ab2bd4173efa7f54 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:36.027279 25092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "90787e1963564cbb8944278903bd2b64"
I20260925 22:45:36.027560 25092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90787e1963564cbb8944278903bd2b64 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:36.029663 25092 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 90787e1963564cbb8944278903bd2b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe55c1f2280d41aabd0d3a40cfddbb1d in term 1.
I20260925 22:45:36.030783 24994 leader_election.cc:302] T 00000000000000000000000000000000 P fe55c1f2280d41aabd0d3a40cfddbb1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90787e1963564cbb8944278903bd2b64, fe55c1f2280d41aabd0d3a40cfddbb1d; no voters:
I20260925 22:45:36.035133 25168 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fe09e14feb0845b5ab2bd4173efa7f54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe55c1f2280d41aabd0d3a40cfddbb1d in term 1.
I20260925 22:45:36.039893 24975 master_runner.cc:263] No leader master found from master fe55c1f2280d41aabd0d3a40cfddbb1d
I20260925 22:45:36.040019 24975 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:45:36.040078 25017 leader_election.cc:288] T 00000000000000000000000000000000 P fe55c1f2280d41aabd0d3a40cfddbb1d [CANDIDATE]: Term 1 election: Requested vote from peers 90787e1963564cbb8944278903bd2b64 (127.2.74.253:40653), fe09e14feb0845b5ab2bd4173efa7f54 (127.2.74.252:38491)
W20260925 22:45:36.028775 25240 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fe55c1f2280d41aabd0d3a40cfddbb1d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:45:36.042320 25240 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:45:36.042517 25229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe55c1f2280d41aabd0d3a40cfddbb1d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:36.042804 25229 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe55c1f2280d41aabd0d3a40cfddbb1d [term 1 LEADER]: Becoming Leader. State: Replica: fe55c1f2280d41aabd0d3a40cfddbb1d, State: Running, Role: LEADER
I20260925 22:45:36.043233 25229 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe55c1f2280d41aabd0d3a40cfddbb1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34285 } } peers { permanent_uuid: "90787e1963564cbb8944278903bd2b64" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 40653 } } peers { permanent_uuid: "fe09e14feb0845b5ab2bd4173efa7f54" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 38491 } }
I20260925 22:45:36.045511 25229 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe55c1f2280d41aabd0d3a40cfddbb1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe55c1f2280d41aabd0d3a40cfddbb1d. Latest consensus state: current_term: 1 leader_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34285 } } peers { permanent_uuid: "90787e1963564cbb8944278903bd2b64" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 40653 } } peers { permanent_uuid: "fe09e14feb0845b5ab2bd4173efa7f54" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 38491 } } }
I20260925 22:45:36.045686 25229 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe55c1f2280d41aabd0d3a40cfddbb1d [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:36.049005 25248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:36.050169 25248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:36.063153 25168 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fe09e14feb0845b5ab2bd4173efa7f54 [term 1 FOLLOWER]: Refusing update from remote peer fe55c1f2280d41aabd0d3a40cfddbb1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:36.064087 25246 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fe55c1f2280d41aabd0d3a40cfddbb1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe09e14feb0845b5ab2bd4173efa7f54" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 38491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:36.069278 25215 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe09e14feb0845b5ab2bd4173efa7f54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe55c1f2280d41aabd0d3a40cfddbb1d. Latest consensus state: current_term: 1 leader_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34285 } } peers { permanent_uuid: "90787e1963564cbb8944278903bd2b64" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 40653 } } peers { permanent_uuid: "fe09e14feb0845b5ab2bd4173efa7f54" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 38491 } } }
I20260925 22:45:36.069438 25215 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe09e14feb0845b5ab2bd4173efa7f54 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:45:36.070894 25092 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 90787e1963564cbb8944278903bd2b64 [term 1 FOLLOWER]: Refusing update from remote peer fe55c1f2280d41aabd0d3a40cfddbb1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:36.072713 25246 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fe55c1f2280d41aabd0d3a40cfddbb1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90787e1963564cbb8944278903bd2b64" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 40653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:36.073536 25120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:36.046403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47448 (local address 127.2.74.253:40653)
0925 22:45:36.046542 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:36.046548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:36.051309 (+ 4761us) server_negotiation.cc:407] Connection header received
0925 22:45:36.063209 (+ 11900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:36.063213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:36.063284 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:36.063346 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:36.064419 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:36.065195 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:36.066458 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:36.066877 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:36.067031 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:36.069572 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:36.069578 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:36.069580 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:36.069618 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:36.071404 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:36.071977 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:36.071981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:36.071985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:36.072087 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:36.072446 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:36.072450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:36.072452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:36.072712 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:36.072813 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:36.073210 (+ 397us) server_negotiation.cc:299] Negotiation successful
0925 22:45:36.073355 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:45:36.077821 25201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90787e1963564cbb8944278903bd2b64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe55c1f2280d41aabd0d3a40cfddbb1d. Latest consensus state: current_term: 1 leader_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34285 } } peers { permanent_uuid: "90787e1963564cbb8944278903bd2b64" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 40653 } } peers { permanent_uuid: "fe09e14feb0845b5ab2bd4173efa7f54" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 38491 } } }
I20260925 22:45:36.078040 25201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90787e1963564cbb8944278903bd2b64 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 22:45:36.078343 25147 tablet.cc:2369] T 00000000000000000000000000000000 P fe09e14feb0845b5ab2bd4173efa7f54: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:45:36.083192 25215 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe09e14feb0845b5ab2bd4173efa7f54 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34285 } } peers { permanent_uuid: "90787e1963564cbb8944278903bd2b64" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 40653 } } peers { permanent_uuid: "fe09e14feb0845b5ab2bd4173efa7f54" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 38491 } } }
I20260925 22:45:36.083359 25215 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe09e14feb0845b5ab2bd4173efa7f54 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:45:36.083843 25229 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe55c1f2280d41aabd0d3a40cfddbb1d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34285 } } peers { permanent_uuid: "90787e1963564cbb8944278903bd2b64" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 40653 } } peers { permanent_uuid: "fe09e14feb0845b5ab2bd4173efa7f54" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 38491 } } }
I20260925 22:45:36.083994 25229 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe55c1f2280d41aabd0d3a40cfddbb1d [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:36.084378 25253 mvcc.cc:205] Tried to move back new op lower bound from 7333381472506097664 to 7333381472437678080. Current Snapshot: MvccSnapshot[applied={T|T < 7333381472506097664}]
W20260925 22:45:36.085930 25071 tablet.cc:2369] T 00000000000000000000000000000000 P 90787e1963564cbb8944278903bd2b64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:45:36.088883 25198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:36.048080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51764 (local address 127.2.74.252:38491)
0925 22:45:36.048210 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:36.048217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:36.055047 (+ 6830us) server_negotiation.cc:407] Connection header received
0925 22:45:36.056694 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:36.056718 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:36.056782 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:36.056843 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:36.060286 (+ 3443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:36.061066 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:36.063703 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:36.064110 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:36.064195 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:36.067391 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:36.067397 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:36.067401 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:36.067437 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:36.069924 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:36.085037 (+ 15113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:36.085042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:36.085044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:36.085170 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:36.086877 (+ 1707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:36.086882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:36.086884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:36.087193 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:36.087359 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:36.088530 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0925 22:45:36.088715 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:45:36.091027 25045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:36.047361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35694 (local address 127.2.74.254:34285)
0925 22:45:36.047484 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:36.047489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:36.054787 (+ 7298us) server_negotiation.cc:407] Connection header received
0925 22:45:36.055306 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:36.055310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:36.055368 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:36.055428 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:36.058099 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:36.058894 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:36.062328 (+ 3434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:36.062694 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:36.062792 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:36.067868 (+ 5076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:36.067873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:36.067877 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:36.067918 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:36.072574 (+ 4656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:36.085550 (+ 12976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:36.085556 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:36.085558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:36.085701 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:36.087750 (+ 2049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:36.087754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:36.087756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:36.088012 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:36.088142 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:36.090675 (+ 2533us) server_negotiation.cc:299] Negotiation successful
0925 22:45:36.090827 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:45:36.092974 25248 catalog_manager.cc:1347] Generated new cluster ID: 35ab3ab1bd2e48bf96e2dba2b7a3b0bb
I20260925 22:45:36.093055 25248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:36.098719 25260 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe55c1f2280d41aabd0d3a40cfddbb1d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34285 } } peers { permanent_uuid: "90787e1963564cbb8944278903bd2b64" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 40653 } } peers { permanent_uuid: "fe09e14feb0845b5ab2bd4173efa7f54" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 38491 } } }
I20260925 22:45:36.098901 25260 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe55c1f2280d41aabd0d3a40cfddbb1d [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:36.100045 25201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90787e1963564cbb8944278903bd2b64 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55c1f2280d41aabd0d3a40cfddbb1d" member_type: VOTER last_known_addr { host: "127.2.74.254" port: 34285 } } peers { permanent_uuid: "90787e1963564cbb8944278903bd2b64" member_type: VOTER last_known_addr { host: "127.2.74.253" port: 40653 } } peers { permanent_uuid: "fe09e14feb0845b5ab2bd4173efa7f54" member_type: VOTER last_known_addr { host: "127.2.74.252" port: 38491 } } }
I20260925 22:45:36.100185 25201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90787e1963564cbb8944278903bd2b64 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:45:36.101987 25255 mvcc.cc:205] Tried to move back new op lower bound from 7333381472506097664 to 7333381472437678080. Current Snapshot: MvccSnapshot[applied={T|T < 7333381472506097664}]
I20260925 22:45:36.144186 25248 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:36.145555 25248 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:36.159013 25248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe55c1f2280d41aabd0d3a40cfddbb1d: Generated new TSK 0
I20260925 22:45:36.159904 25248 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:45:36.186677 25045 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:36.184806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35696 (local address 127.2.74.254:34285)
0925 22:45:36.184979 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:36.184985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:36.185433 (+ 448us) server_negotiation.cc:407] Connection header received
0925 22:45:36.185671 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:36.185675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:36.185732 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:36.185789 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:36.186530 (+ 741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35696: BlockingRecv error: recv got EOF from 127.0.0.1:35696 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260925 22:45:36.186678 25120 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:36.185097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47454 (local address 127.2.74.253:40653)
0925 22:45:36.185383 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:36.185388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:36.185618 (+ 230us) server_negotiation.cc:407] Connection header received
0925 22:45:36.185825 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:36.185828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:36.185888 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:36.185966 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:36.186519 (+ 553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47454: BlockingRecv error: recv got EOF from 127.0.0.1:47454 (error 108)
Metrics: {"server-negotiator.queue_time_us":191}
W20260925 22:45:36.187095 25198 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:36.185229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51774 (local address 127.2.74.252:38491)
0925 22:45:36.185381 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:36.185386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:36.186036 (+ 650us) server_negotiation.cc:407] Connection header received
0925 22:45:36.186213 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:36.186218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:36.186275 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:36.186330 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:36.186977 (+ 647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51774: BlockingRecv error: recv got EOF from 127.0.0.1:51774 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260925 22:45:36.190882 25120 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:36.189451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47466 (local address 127.2.74.253:40653)
0925 22:45:36.189575 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:36.189578 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:36.189693 (+ 115us) server_negotiation.cc:407] Connection header received
0925 22:45:36.189852 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:36.189855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:36.189923 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:36.189994 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:36.190730 (+ 736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47466: BlockingRecv error: recv got EOF from 127.0.0.1:47466 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 22:45:36.195604 25120 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:36.194607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47482 (local address 127.2.74.253:40653)
0925 22:45:36.194751 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:36.194755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:36.194768 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:45:36.194816 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:36.194818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:36.194862 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:36.194989 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:36.195466 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47482: BlockingRecv error: recv got EOF from 127.0.0.1:47482 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 22:45:36.201380 25120 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:36.200154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47496 (local address 127.2.74.253:40653)
0925 22:45:36.200299 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:36.200304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:36.200415 (+ 111us) server_negotiation.cc:407] Connection header received
0925 22:45:36.200591 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:36.200595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:36.200653 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:36.200745 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:36.201226 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47496: BlockingRecv error: recv got EOF from 127.0.0.1:47496 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 22:45:36.208994 25120 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:36.207630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47502 (local address 127.2.74.253:40653)
0925 22:45:36.207818 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:36.207823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:36.208034 (+ 211us) server_negotiation.cc:407] Connection header received
0925 22:45:36.208207 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:36.208211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:36.208309 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:36.208398 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:36.208864 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47502: BlockingRecv error: recv got EOF from 127.0.0.1:47502 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260925 22:45:36.220589 25120 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:36.219199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47506 (local address 127.2.74.253:40653)
0925 22:45:36.219359 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:36.219364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:36.219549 (+ 185us) server_negotiation.cc:407] Connection header received
0925 22:45:36.219720 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:36.219723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:36.219776 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:36.219851 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:36.220460 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47506: BlockingRecv error: recv got EOF from 127.0.0.1:47506 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 22:45:36.240341 25120 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:36.239052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47520 (local address 127.2.74.253:40653)
0925 22:45:36.239196 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:36.239201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:36.239344 (+ 143us) server_negotiation.cc:407] Connection header received
0925 22:45:36.239534 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:36.239537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:36.239593 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:36.239674 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:36.240202 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47520: BlockingRecv error: recv got EOF from 127.0.0.1:47520 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 22:45:36.275905 25120 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:36.274510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47536 (local address 127.2.74.253:40653)
0925 22:45:36.274689 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:36.274694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:36.274827 (+ 133us) server_negotiation.cc:407] Connection header received
0925 22:45:36.275098 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:36.275102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:36.275165 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:36.275283 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:36.275789 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47536: BlockingRecv error: recv got EOF from 127.0.0.1:47536 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260925 22:45:36.343628 25120 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:36.342278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47548 (local address 127.2.74.253:40653)
0925 22:45:36.342447 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:36.342452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:36.342649 (+ 197us) server_negotiation.cc:407] Connection header received
0925 22:45:36.342834 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:36.342845 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:36.342896 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:36.343038 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:36.343506 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47548: BlockingRecv error: recv got EOF from 127.0.0.1:47548 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260925 22:45:36.474874 25120 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:36.473749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47550 (local address 127.2.74.253:40653)
0925 22:45:36.473892 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:36.473896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:36.474059 (+ 163us) server_negotiation.cc:407] Connection header received
0925 22:45:36.474183 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:36.474185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:36.474233 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:36.474318 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:36.474729 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47550: BlockingRecv error: recv got EOF from 127.0.0.1:47550 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 22:45:36.734223 25120 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:36.733133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47560 (local address 127.2.74.253:40653)
0925 22:45:36.733300 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:36.733304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:36.733410 (+ 106us) server_negotiation.cc:407] Connection header received
0925 22:45:36.733530 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:36.733533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:36.733581 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:36.733664 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:36.734076 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47560: BlockingRecv error: recv got EOF from 127.0.0.1:47560 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 22:45:36.891247 25212 catalog_manager.cc:1259] Loaded cluster ID: 35ab3ab1bd2e48bf96e2dba2b7a3b0bb
I20260925 22:45:36.891340 25212 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 90787e1963564cbb8944278903bd2b64: loading cluster ID for follower catalog manager: success
I20260925 22:45:36.893071 25212 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 90787e1963564cbb8944278903bd2b64: acquiring CA information for follower catalog manager: success
I20260925 22:45:36.894232 25212 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 90787e1963564cbb8944278903bd2b64: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:45:36.904785 25226 catalog_manager.cc:1259] Loaded cluster ID: 35ab3ab1bd2e48bf96e2dba2b7a3b0bb
I20260925 22:45:36.904927 25226 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fe09e14feb0845b5ab2bd4173efa7f54: loading cluster ID for follower catalog manager: success
I20260925 22:45:36.906675 25226 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fe09e14feb0845b5ab2bd4173efa7f54: acquiring CA information for follower catalog manager: success
I20260925 22:45:36.908071 25226 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fe09e14feb0845b5ab2bd4173efa7f54: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:45:37.252807 25326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:37.248662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47564 (local address 127.2.74.253:40653)
0925 22:45:37.248949 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:37.248956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:37.248975 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:45:37.249168 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:37.249171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:37.249311 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:37.249437 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:45:37.249701 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:37.250242 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:37.251205 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:37.251444 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:37.251568 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:37.251916 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:45:37.252371 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:45:37.252519 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:37.252573 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 22:45:37.252638 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":101,"threads_started":1}
I20260925 22:45:37.260260 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:37.256219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51778 (local address 127.2.74.252:38491)
0925 22:45:37.256491 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:37.256495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:37.256509 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:45:37.256823 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:37.256826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:37.256954 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:37.257069 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:45:37.257307 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:37.257857 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:37.258691 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:37.258909 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:37.259094 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:37.259365 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:45:37.259826 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:45:37.259925 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:37.259983 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 22:45:37.260040 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":67,"threads_started":1}
I20260925 22:45:37.268350 25339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:37.263772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35706 (local address 127.2.74.254:34285)
0925 22:45:37.264450 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:37.264455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:37.264468 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:45:37.264578 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:37.264581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:37.264754 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:37.264906 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:45:37.265250 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:37.265796 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:37.266726 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:37.267030 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:37.267144 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:37.267500 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:45:37.267868 (+ 368us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:45:37.267949 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:37.268109 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 22:45:37.268173 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":77,"threads_started":1}
I20260925 22:45:37.268821 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:37.264296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51786 (local address 127.2.74.252:38491)
0925 22:45:37.264440 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:37.264444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:37.264572 (+ 128us) server_negotiation.cc:407] Connection header received
0925 22:45:37.264672 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:37.264678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:37.264737 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:37.264809 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:45:37.265103 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:37.265872 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:37.267487 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:37.267799 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:37.267922 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:37.268310 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:45:37.268406 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:45:37.268452 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:37.268619 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 22:45:37.268686 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:45:37.268956 25326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:37.263843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47576 (local address 127.2.74.253:40653)
0925 22:45:37.263994 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:37.264001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:37.264239 (+ 238us) server_negotiation.cc:407] Connection header received
0925 22:45:37.264443 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:37.264446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:37.264506 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:37.264596 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:45:37.264961 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:37.265829 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:37.267464 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:37.267721 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:37.267794 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:37.268486 (+ 692us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:45:37.268580 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:45:37.268657 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:37.268746 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 22:45:37.268805 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:45:37.272146 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 24975
I20260925 22:45:37.279227 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 25050
I20260925 22:45:37.285847 2347 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4wBtU/build/debug/bin/kudu with pid 25126
2026-09-25T22:45:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3315 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6524 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (289696 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST